[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:51:13 dist-test-slave-1x4g krb5kdc[13913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:51:13 dist-test-slave-1x4g krb5kdc[13913](info): set up 2 sockets
Aug 06 04:51:13 dist-test-slave-1x4g krb5kdc[13913](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 04:51:14.026172 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:51:14 dist-test-slave-1x4g krb5kdc[13913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:51:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:51:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:51:14.182756 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:34515
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:34515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:14.578864 13926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:14.579780 13926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:14.579905 13926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:14.586153 13926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:51:14.586249 13926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:14.586290 13926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:14.586335 13926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:51:14.586395 13926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:51:14.594696 13926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:34515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:34515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:14.597813 13926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:14.599890 13926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:14.618687 13931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:14.618655 13932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:14.619135 13934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:14.619478 13926 server_base.cc:1034] running on GCE node
I20260806 04:51:14.620323 13926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:14.621713 13926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:14.622907 13926 hybrid_clock.cc:648] HybridClock initialized: now 1785991874622886 us; error 48 us; skew 500 ppm
I20260806 04:51:14.623983 13926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:14 dist-test-slave-1x4g krb5kdc[13913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991874, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:14.627065 13926 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:51:14.628546 13926 webserver.cc:460] Webserver started at http://127.13.148.190:33733/ using document root <none> and password file <none>
I20260806 04:51:14.630178 13926 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:14.630275 13926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:14.630523 13926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:14.637646 13926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "f526777e5d5f42949336805aaf39e703"
format_stamp: "Formatted at 2026-08-06 04:51:14 on dist-test-slave-1x4g"
server_key: "85f49da134c198c11ab451377b8dca8f"
server_key_iv: "8ef582af489a0032e1a9900e09aaa00c"
server_key_version: "encryptionkey@0"
I20260806 04:51:14.638336 13926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "f526777e5d5f42949336805aaf39e703"
format_stamp: "Formatted at 2026-08-06 04:51:14 on dist-test-slave-1x4g"
server_key: "85f49da134c198c11ab451377b8dca8f"
server_key_iv: "8ef582af489a0032e1a9900e09aaa00c"
server_key_version: "encryptionkey@0"
I20260806 04:51:14.644105 13926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 04:51:14.651024 13941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:14.652256 13926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:51:14.652383 13926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "f526777e5d5f42949336805aaf39e703"
format_stamp: "Formatted at 2026-08-06 04:51:14 on dist-test-slave-1x4g"
server_key: "85f49da134c198c11ab451377b8dca8f"
server_key_iv: "8ef582af489a0032e1a9900e09aaa00c"
server_key_version: "encryptionkey@0"
I20260806 04:51:14.652536 13926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:14.678486 13926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:14.681993 13926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:14.682209 13926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:14.703174 13926 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:34515
I20260806 04:51:14.703195 13992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:34515 every 8 connection(s)
I20260806 04:51:14.705348 13926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:51:14.711035 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 13926
I20260806 04:51:14.711319 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:51:14.713233 13906 external_mini_cluster.cc:1428] Setting key afdeb78b1eebb2eb309e7b1d51a7e0a5
I20260806 04:51:14.713286 13993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:14.728184 13993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703: Bootstrap starting.
I20260806 04:51:14.732177 13993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:14.733721 13993 log.cc:824] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:14.737054 13993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703: No bootstrap required, opened a new log
Aug 06 04:51:14 dist-test-slave-1x4g krb5kdc[13913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:14.749790 13993 raft_consensus.cc:348] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f526777e5d5f42949336805aaf39e703" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34515 } }
I20260806 04:51:14.750212 13993 raft_consensus.cc:374] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:14.750311 13993 raft_consensus.cc:729] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f526777e5d5f42949336805aaf39e703, State: Initialized, Role: FOLLOWER
I20260806 04:51:14.750905 13993 consensus_queue.cc:260] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f526777e5d5f42949336805aaf39e703" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34515 } }
I20260806 04:51:14.751163 13993 raft_consensus.cc:388] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:51:14.751526 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:14.719754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54448 (local address 127.13.148.190:34515)
0806 04:51:14.720533 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:14.720959 (+ 426us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:14.721779 (+ 820us) server_negotiation.cc:407] Connection header received
0806 04:51:14.730786 (+ 9007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:14.730808 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:14.731218 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:14.731620 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:14.734349 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:14.736025 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:14.737800 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:14.738312 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:14.738488 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:14.743263 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:14.743271 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:14.743286 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:14.743330 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:14.746987 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:14.747738 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:14.747746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:14.747754 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:14.747959 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:14.748331 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:14.748335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:14.748338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:14.748839 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:14.749046 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:14.749826 (+ 780us) server_negotiation.cc:299] Negotiation successful
0806 04:51:14.750181 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":330,"threads_started":1}
I20260806 04:51:14.751286 13993 raft_consensus.cc:482] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:51:14.752344 13993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:14.754724 13993 raft_consensus.cc:504] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f526777e5d5f42949336805aaf39e703" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34515 } }
I20260806 04:51:14.756253 13993 leader_election.cc:302] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f526777e5d5f42949336805aaf39e703; no voters:
I20260806 04:51:14.757004 13993 leader_election.cc:288] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:51:14.757130 13998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:14.758160 13998 raft_consensus.cc:686] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [term 1 LEADER]: Becoming Leader. State: Replica: f526777e5d5f42949336805aaf39e703, State: Running, Role: LEADER
I20260806 04:51:14.758702 13998 consensus_queue.cc:237] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f526777e5d5f42949336805aaf39e703" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34515 } }
I20260806 04:51:14.761348 13993 sys_catalog.cc:564] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 04:51:14.773835 14013 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:51:14.773924 14013 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:51:14.775092 14000 sys_catalog.cc:455] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f526777e5d5f42949336805aaf39e703. Latest consensus state: current_term: 1 leader_uuid: "f526777e5d5f42949336805aaf39e703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f526777e5d5f42949336805aaf39e703" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34515 } } }
I20260806 04:51:14.775139 13999 sys_catalog.cc:455] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f526777e5d5f42949336805aaf39e703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f526777e5d5f42949336805aaf39e703" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34515 } } }
I20260806 04:51:14.775364 14000 sys_catalog.cc:458] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:14.775406 13999 sys_catalog.cc:458] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:14.776198 14015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:51:14.778010 14015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:51:14.785501 14015 catalog_manager.cc:1347] Generated new cluster ID: 41bccabd03864ded80bfd4fbce78b2eb
I20260806 04:51:14.785605 14015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:51:14.820751 14015 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:51:14.822460 14015 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:51:14.839902 14015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f526777e5d5f42949336805aaf39e703: Generated new TSK 0
I20260806 04:51:14.840916 14015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:51:14.882647 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:34515
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:15.199182 14019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:15.199421 14019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:15.199754 14019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:15.206393 14019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:15.206480 14019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:15.206651 14019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:51:15.215507 14019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:34515
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:15.216717 14019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:15.218037 14019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:15.231527 14024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:15.231506 14027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:15.231459 14025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:15.232125 14019 server_base.cc:1034] running on GCE node
I20260806 04:51:15.232590 14019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:15.233398 14019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:15.234627 14019 hybrid_clock.cc:648] HybridClock initialized: now 1785991875234595 us; error 52 us; skew 500 ppm
I20260806 04:51:15.234907 14019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:15 dist-test-slave-1x4g krb5kdc[13913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991875, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:15.237915 14019 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:51:15.239295 14019 webserver.cc:460] Webserver started at http://127.13.148.129:36037/ using document root <none> and password file <none>
I20260806 04:51:15.239988 14019 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:15.240123 14019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:15.240413 14019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:15.243156 14019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "5b363e16f7d44304ba88ed8ec58d8355"
format_stamp: "Formatted at 2026-08-06 04:51:15 on dist-test-slave-1x4g"
server_key: "098906e8eba53fe16fff69de75c6ba7b"
server_key_iv: "016543b00536cd49357015f938d5294c"
server_key_version: "encryptionkey@0"
I20260806 04:51:15.243899 14019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "5b363e16f7d44304ba88ed8ec58d8355"
format_stamp: "Formatted at 2026-08-06 04:51:15 on dist-test-slave-1x4g"
server_key: "098906e8eba53fe16fff69de75c6ba7b"
server_key_iv: "016543b00536cd49357015f938d5294c"
server_key_version: "encryptionkey@0"
I20260806 04:51:15.248521 14019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260806 04:51:15.251816 14034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:15.253075 14019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:51:15.253222 14019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "5b363e16f7d44304ba88ed8ec58d8355"
format_stamp: "Formatted at 2026-08-06 04:51:15 on dist-test-slave-1x4g"
server_key: "098906e8eba53fe16fff69de75c6ba7b"
server_key_iv: "016543b00536cd49357015f938d5294c"
server_key_version: "encryptionkey@0"
I20260806 04:51:15.253393 14019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:15.278595 14019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:15.282292 14019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:15.282539 14019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:15.283351 14019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:15.284569 14019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:15.284667 14019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:15.284849 14019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:15.284938 14019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 04:51:15.318055 14019 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:41609
I20260806 04:51:15.318137 14149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:41609 every 8 connection(s)
I20260806 04:51:15.319352 14019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:51:15.322845 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 14019
I20260806 04:51:15.323086 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:51:15.323978 13906 external_mini_cluster.cc:1428] Setting key 23a32cc2c18f15cb45d543f45fec9051
Aug 06 04:51:15 dist-test-slave-1x4g krb5kdc[13913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991875, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
I20260806 04:51:15.342387 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:15.323572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:42941 (local address 127.13.148.190:34515)
0806 04:51:15.324099 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:15.324105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:15.325068 (+ 963us) server_negotiation.cc:407] Connection header received
0806 04:51:15.327289 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:15.327293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:15.327353 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:15.327449 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:15.329630 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:15.330276 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:15.332039 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:15.332423 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:15.332544 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:15.335786 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:15.335794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:15.335800 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:15.335859 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:15.339469 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:15.340149 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:15.340153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:15.340155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:15.340266 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:15.340673 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:15.340682 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:15.340684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:15.340996 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:15.341148 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:15.341795 (+ 647us) server_negotiation.cc:299] Negotiation successful
0806 04:51:15.342007 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":244,"threads_started":1}
I20260806 04:51:15.343222 14153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:15.324008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:34515 (local address 127.13.148.129:42941)
0806 04:51:15.324882 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:15.324944 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:15.327009 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:15.327741 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:15.327767 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:15.328539 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:15.329369 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:15.329396 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:15.330425 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:15.330432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:15.331895 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:15.331906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:15.332640 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:15.332648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:15.332940 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:15.333599 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:15.333619 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:15.335583 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:15.339621 (+ 4038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:15.339634 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:15.339644 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:15.340009 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:15.340381 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:15.340385 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:15.340389 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:15.340555 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:15.341126 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:15.341133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:15.341466 (+ 333us) client_negotiation.cc:769] Sending connection context
0806 04:51:15.341702 (+ 236us) client_negotiation.cc:240] Negotiation successful
0806 04:51:15.342162 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":365,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
I20260806 04:51:15.344753 14150 heartbeater.cc:347] Connected to a master server at 127.13.148.190:34515
I20260806 04:51:15.345069 14150 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:15.345978 14150 heartbeater.cc:511] Master 127.13.148.190:34515 requested a full tablet report, sending...
I20260806 04:51:15.349594 13958 ts_manager.cc:194] Registered new tserver with Master: 5b363e16f7d44304ba88ed8ec58d8355 (127.13.148.129:41609)
I20260806 04:51:15.352910 13958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:42941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:51:15.364051 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:34515
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:15.676517 14156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:15.676913 14156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:15.677105 14156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:15.683573 14156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:15.683665 14156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:15.683809 14156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:51:15.692199 14156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:34515
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:15.693321 14156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:15.694563 14156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:15.708415 14163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:15.708557 14162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:15.708750 14165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:15.709585 14156 server_base.cc:1034] running on GCE node
I20260806 04:51:15.710167 14156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:15.711212 14156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:15.712435 14156 hybrid_clock.cc:648] HybridClock initialized: now 1785991875712406 us; error 35 us; skew 500 ppm
I20260806 04:51:15.712729 14156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:15 dist-test-slave-1x4g krb5kdc[13913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991875, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:15.716114 14156 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:51:15.717819 14156 webserver.cc:460] Webserver started at http://127.13.148.130:43185/ using document root <none> and password file <none>
I20260806 04:51:15.718468 14156 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:15.718609 14156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:15.718966 14156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:15.721855 14156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "5eec91df19454779a79f27aea1fe6f8d"
format_stamp: "Formatted at 2026-08-06 04:51:15 on dist-test-slave-1x4g"
server_key: "e2f8a4c1491569b3aede86a1a7bb51fb"
server_key_iv: "a714b047fdf6d6ac306b5cad3995e840"
server_key_version: "encryptionkey@0"
I20260806 04:51:15.722618 14156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "5eec91df19454779a79f27aea1fe6f8d"
format_stamp: "Formatted at 2026-08-06 04:51:15 on dist-test-slave-1x4g"
server_key: "e2f8a4c1491569b3aede86a1a7bb51fb"
server_key_iv: "a714b047fdf6d6ac306b5cad3995e840"
server_key_version: "encryptionkey@0"
I20260806 04:51:15.727439 14156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260806 04:51:15.730706 14172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:15.731909 14156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:51:15.732082 14156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "5eec91df19454779a79f27aea1fe6f8d"
format_stamp: "Formatted at 2026-08-06 04:51:15 on dist-test-slave-1x4g"
server_key: "e2f8a4c1491569b3aede86a1a7bb51fb"
server_key_iv: "a714b047fdf6d6ac306b5cad3995e840"
server_key_version: "encryptionkey@0"
I20260806 04:51:15.732267 14156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:15.745036 14156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:15.748719 14156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:15.749035 14156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:15.749883 14156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:15.751096 14156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:15.751204 14156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:15.751360 14156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:15.751446 14156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:15.784874 14156 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:45865
I20260806 04:51:15.784929 14287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:45865 every 8 connection(s)
I20260806 04:51:15.786110 14156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:51:15.788514 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 14156
I20260806 04:51:15.788699 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:51:15.789161 13906 external_mini_cluster.cc:1428] Setting key c8d28eeb633f439984f4ac8b8d917bd1
Aug 06 04:51:15 dist-test-slave-1x4g krb5kdc[13913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991875, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
I20260806 04:51:15.806684 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:15.788703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:37667 (local address 127.13.148.190:34515)
0806 04:51:15.788913 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:15.788919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:15.790209 (+ 1290us) server_negotiation.cc:407] Connection header received
0806 04:51:15.792166 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:15.792174 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:15.792266 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:15.792379 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:15.793907 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:15.794952 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:15.796459 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:15.797054 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:15.797210 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:15.800982 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:15.800986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:15.800990 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:15.801025 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:15.803612 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:15.804442 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:15.804446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:15.804448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:15.804566 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:15.805093 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:15.805096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:15.805098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:15.805436 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:15.805561 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:15.806174 (+ 613us) server_negotiation.cc:299] Negotiation successful
0806 04:51:15.806414 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 04:51:15.807458 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:15.789218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:34515 (local address 127.13.148.130:37667)
0806 04:51:15.790045 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:15.790093 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:15.791896 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:15.792582 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:15.792590 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:15.793208 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:15.793741 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:15.793759 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:15.795107 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:15.795111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:15.796280 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:15.796293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:15.797308 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:15.797318 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:15.797568 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:15.798265 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:15.798292 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:15.800745 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:15.803750 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:15.803760 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:15.803775 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:15.804317 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:15.804697 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:15.804703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:15.804707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:15.804969 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:15.805567 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:15.805578 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:15.805858 (+ 280us) client_negotiation.cc:769] Sending connection context
0806 04:51:15.806086 (+ 228us) client_negotiation.cc:240] Negotiation successful
0806 04:51:15.806445 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":333,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
I20260806 04:51:15.809058 14288 heartbeater.cc:347] Connected to a master server at 127.13.148.190:34515
I20260806 04:51:15.809372 14288 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:15.810074 14288 heartbeater.cc:511] Master 127.13.148.190:34515 requested a full tablet report, sending...
I20260806 04:51:15.811545 13958 ts_manager.cc:194] Registered new tserver with Master: 5eec91df19454779a79f27aea1fe6f8d (127.13.148.130:45865)
I20260806 04:51:15.812295 13958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:37667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
I20260806 04:51:15.825629 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:34515
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:16.135699 14293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:16.135951 14293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:16.136113 14293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:16.142432 14293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:16.142524 14293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:16.142657 14293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:51:16.151402 14293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:34515
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:16.152534 14293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:16.153841 14293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:16.166349 14298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:16.166332 14299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:16.166904 14301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:16.167834 14293 server_base.cc:1034] running on GCE node
I20260806 04:51:16.168349 14293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:16.169257 14293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:16.170480 14293 hybrid_clock.cc:648] HybridClock initialized: now 1785991876170444 us; error 54 us; skew 500 ppm
I20260806 04:51:16.170800 14293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:16 dist-test-slave-1x4g krb5kdc[13913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991876, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:16.173822 14293 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:51:16.175118 14293 webserver.cc:460] Webserver started at http://127.13.148.131:42947/ using document root <none> and password file <none>
I20260806 04:51:16.175725 14293 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:16.175859 14293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:16.176146 14293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:16.178936 14293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "d12f0125d92c475887faf9ca0434f077"
format_stamp: "Formatted at 2026-08-06 04:51:16 on dist-test-slave-1x4g"
server_key: "bf1196ded5fbf67101b697eac77d2a69"
server_key_iv: "ca139c8ae24853e9ab34d1ca90672902"
server_key_version: "encryptionkey@0"
I20260806 04:51:16.179644 14293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "d12f0125d92c475887faf9ca0434f077"
format_stamp: "Formatted at 2026-08-06 04:51:16 on dist-test-slave-1x4g"
server_key: "bf1196ded5fbf67101b697eac77d2a69"
server_key_iv: "ca139c8ae24853e9ab34d1ca90672902"
server_key_version: "encryptionkey@0"
I20260806 04:51:16.184098 14293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260806 04:51:16.187525 14308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:16.188680 14293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:51:16.188843 14293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "d12f0125d92c475887faf9ca0434f077"
format_stamp: "Formatted at 2026-08-06 04:51:16 on dist-test-slave-1x4g"
server_key: "bf1196ded5fbf67101b697eac77d2a69"
server_key_iv: "ca139c8ae24853e9ab34d1ca90672902"
server_key_version: "encryptionkey@0"
I20260806 04:51:16.189041 14293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:16.201333 14293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:16.204676 14293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:16.204953 14293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:16.205771 14293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:16.206990 14293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:16.207094 14293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:16.207234 14293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:16.207312 14293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:16.238435 14293 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:33581
I20260806 04:51:16.238500 14423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:33581 every 8 connection(s)
I20260806 04:51:16.239756 14293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:51:16.249979 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 14293
I20260806 04:51:16.250190 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:51:16.250737 13906 external_mini_cluster.cc:1428] Setting key 953bbcf4ffd1dc5b2b9cbdc0ed570043
Aug 06 04:51:16 dist-test-slave-1x4g krb5kdc[13913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991876, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:16.258371 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:16.242312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:38331 (local address 127.13.148.190:34515)
0806 04:51:16.242495 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:16.242499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:16.243547 (+ 1048us) server_negotiation.cc:407] Connection header received
0806 04:51:16.245167 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:16.245173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:16.245252 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:16.245347 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:16.246673 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:16.247402 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:16.248846 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:16.249280 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:16.249406 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:16.252986 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:16.252991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:16.252996 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:16.253034 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:16.255578 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:16.256252 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:16.256255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:16.256258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:16.256372 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:16.256888 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:16.256891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:16.256893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:16.257141 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:16.257251 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:16.257988 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 04:51:16.258145 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:51:16.259315 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:16.242792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:34515 (local address 127.13.148.131:38331)
0806 04:51:16.243402 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:16.243442 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:16.244925 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:16.245557 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:16.245564 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:16.246019 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:16.246502 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:16.246516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:16.247574 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:16.247578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:16.248699 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:16.248710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:16.249482 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:16.249490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:16.249777 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:16.250466 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:16.250493 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:16.252819 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:16.255748 (+ 2929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:16.255754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:16.255763 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:16.256149 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:16.256463 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:16.256466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:16.256468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:16.256719 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:16.257246 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:16.257251 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:16.257576 (+ 325us) client_negotiation.cc:769] Sending connection context
0806 04:51:16.257902 (+ 326us) client_negotiation.cc:240] Negotiation successful
0806 04:51:16.258270 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":236,"threads_started":1}
I20260806 04:51:16.260504 14424 heartbeater.cc:347] Connected to a master server at 127.13.148.190:34515
I20260806 04:51:16.260743 14424 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:16.261473 14424 heartbeater.cc:511] Master 127.13.148.190:34515 requested a full tablet report, sending...
I20260806 04:51:16.262748 13958 ts_manager.cc:194] Registered new tserver with Master: d12f0125d92c475887faf9ca0434f077 (127.13.148.131:33581)
I20260806 04:51:16.263530 13958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:38331
I20260806 04:51:16.266320 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 04:51:16 dist-test-slave-1x4g krb5kdc[13913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991876, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 04:51:16 dist-test-slave-1x4g krb5kdc[13913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991876, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:51:16.308070 14431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:16.294655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.13.148.129:41609)
0806 04:51:16.295207 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:16.295215 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:16.295315 (+ 100us) server_negotiation.cc:407] Connection header received
0806 04:51:16.295449 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:16.295455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:16.295667 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:16.295777 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:16.297191 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:16.298162 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:16.299244 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:16.299708 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:16.299791 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:16.303291 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:16.303301 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:16.303312 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:16.303340 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:16.305773 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:16.306276 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:16.306282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:16.306284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:16.306389 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:16.306726 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:16.306729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:16.306731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:16.307046 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:16.307263 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:16.307490 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 04:51:16.307671 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":279,"threads_started":1}
W20260806 04:51:16.309293 14063 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40070
Aug 06 04:51:16 dist-test-slave-1x4g krb5kdc[13913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 04:51:16 dist-test-slave-1x4g krb5kdc[13913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:51:16.350562 14431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:16.337402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.13.148.129:41609)
0806 04:51:16.337602 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:16.337611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:16.337949 (+ 338us) server_negotiation.cc:407] Connection header received
0806 04:51:16.338164 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:16.338168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:16.338252 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:16.338356 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:16.339653 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:16.340612 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:16.341844 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:16.342219 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:16.342340 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:16.345869 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:16.345876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:16.345884 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:16.345911 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:16.348154 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:16.348686 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:16.348692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:16.348695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:16.348858 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:16.349223 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:16.349227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:16.349230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:16.349507 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:16.349737 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:16.350022 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 04:51:16.350237 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:51:16.351874 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 14019
I20260806 04:51:16.355684 14150 heartbeater.cc:503] Master 127.13.148.190:34515 was elected leader, sending a full tablet report...
I20260806 04:51:16.363812 14431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:16.352637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.13.148.129:41609)
0806 04:51:16.352872 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:16.352879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:16.353148 (+ 269us) server_negotiation.cc:407] Connection header received
0806 04:51:16.353300 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:16.353309 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:16.353354 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:16.353448 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:16.354517 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:16.355115 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:16.356408 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:16.356899 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:16.357045 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:16.359164 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:16.359168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:16.359171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:16.359196 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:16.361571 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:16.362262 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:16.362267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:16.362269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:16.362413 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:16.362812 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:16.362815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:16.362816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:16.363017 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:16.363156 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:16.363365 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 04:51:16.363492 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 04:51:16.364426 14145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:16.442071 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 14019
I20260806 04:51:16.459414 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 14156
Aug 06 04:51:16 dist-test-slave-1x4g krb5kdc[13913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:51:16.473927 14438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:16.460116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.13.148.130:45865)
0806 04:51:16.460592 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:16.460599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:16.460622 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:51:16.460679 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:16.460684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:16.460965 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:16.461099 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:16.462248 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:16.463283 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:16.464497 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:16.465082 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:16.465188 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:16.468896 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:16.468906 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:16.468919 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:16.468946 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:16.471383 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:16.471985 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:16.471992 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:16.471994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:16.472114 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:16.472478 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:16.472481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:16.472483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:16.473047 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:16.473251 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:16.473511 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 04:51:16.473676 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":238,"threads_started":1}
I20260806 04:51:16.475071 14283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:16.549957 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 14156
I20260806 04:51:16.571244 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 14293
Aug 06 04:51:16 dist-test-slave-1x4g krb5kdc[13913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
I20260806 04:51:16.585638 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:16.572238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40306 (local address 127.13.148.131:33581)
0806 04:51:16.572887 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:16.572894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:16.572923 (+ 29us) server_negotiation.cc:407] Connection header received
0806 04:51:16.572981 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:16.572987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:16.573202 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:16.573310 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:16.574389 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:16.575338 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:16.576502 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:16.577140 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:16.577233 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:16.580707 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:16.580717 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:16.580729 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:16.580756 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:16.583327 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:16.583886 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:16.583892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:16.583894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:16.584003 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:16.584278 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:16.584281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:16.584283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:16.584580 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:16.584769 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:16.585102 (+ 333us) server_negotiation.cc:299] Negotiation successful
0806 04:51:16.585306 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":315,"threads_started":1}
I20260806 04:51:16.586830 14419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:16.662899 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 14293
I20260806 04:51:16.680662 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 13926
I20260806 04:51:16.681823 13988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:16.780604 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 13926
2026-08-06T04:51:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3757 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:51:16 dist-test-slave-1x4g krb5kdc[14445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:51:16 dist-test-slave-1x4g krb5kdc[14445](info): set up 2 sockets
Aug 06 04:51:16 dist-test-slave-1x4g krb5kdc[14445](info): commencing operation
krb5kdc: starting...
W20260806 04:51:17.761260 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:51:17 dist-test-slave-1x4g krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:51:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:51:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:51:17.915294 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41181
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:41181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:18.201995 14458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:18.202302 14458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:18.202461 14458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:18.209117 14458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:51:18.209257 14458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:18.209355 14458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:18.209457 14458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:51:18.209537 14458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:51:18.218117 14458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:41181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:18.219348 14458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:18.220532 14458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:18.231384 14465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:18.231465 14464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:18.231426 14467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:18.232085 14458 server_base.cc:1034] running on GCE node
I20260806 04:51:18.232890 14458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:18.234069 14458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:18.235278 14458 hybrid_clock.cc:648] HybridClock initialized: now 1785991878235249 us; error 41 us; skew 500 ppm
I20260806 04:51:18.235594 14458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:18 dist-test-slave-1x4g krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991878, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:18.243233 14458 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:51:18.244702 14458 webserver.cc:460] Webserver started at http://127.13.148.190:45849/ using document root <none> and password file <none>
I20260806 04:51:18.245352 14458 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:18.245489 14458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:18.245842 14458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:18.248620 14458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "ea823db379f9475fa5da383cb5036182"
format_stamp: "Formatted at 2026-08-06 04:51:18 on dist-test-slave-1x4g"
server_key: "9432228829a901cb8b646255b5fb8623"
server_key_iv: "a7423212a2f4204223e454edad22ea1f"
server_key_version: "encryptionkey@0"
I20260806 04:51:18.249341 14458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "ea823db379f9475fa5da383cb5036182"
format_stamp: "Formatted at 2026-08-06 04:51:18 on dist-test-slave-1x4g"
server_key: "9432228829a901cb8b646255b5fb8623"
server_key_iv: "a7423212a2f4204223e454edad22ea1f"
server_key_version: "encryptionkey@0"
I20260806 04:51:18.254007 14458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 04:51:18.257072 14474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:18.258282 14458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:51:18.258445 14458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "ea823db379f9475fa5da383cb5036182"
format_stamp: "Formatted at 2026-08-06 04:51:18 on dist-test-slave-1x4g"
server_key: "9432228829a901cb8b646255b5fb8623"
server_key_iv: "a7423212a2f4204223e454edad22ea1f"
server_key_version: "encryptionkey@0"
I20260806 04:51:18.258625 14458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:18.284158 14458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:18.288137 14458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:18.288415 14458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:18.304939 14458 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:41181
I20260806 04:51:18.304932 14525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:41181 every 8 connection(s)
I20260806 04:51:18.306166 14458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:51:18.309013 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 14458
I20260806 04:51:18.309257 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:51:18.310035 14526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:18.310086 13906 external_mini_cluster.cc:1428] Setting key be1808a203832be1a14e487f9fd1ac09
I20260806 04:51:18.323321 14526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182: Bootstrap starting.
Aug 06 04:51:18 dist-test-slave-1x4g krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:18.326977 14526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:18.328287 14526 log.cc:824] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:18.331207 14526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182: No bootstrap required, opened a new log
I20260806 04:51:18.332095 14529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:18.311937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51180 (local address 127.13.148.190:41181)
0806 04:51:18.312665 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:18.312677 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:18.312732 (+ 55us) server_negotiation.cc:407] Connection header received
0806 04:51:18.314445 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:18.314465 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:18.314814 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:18.315113 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:18.316888 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:18.318144 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:18.319692 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:18.320284 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:18.320399 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:18.324386 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:18.324395 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:18.324411 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:18.324464 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:18.328120 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:18.328755 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:18.328766 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:18.328800 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:18.329000 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:18.329382 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:18.329385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:18.329387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:18.329950 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:18.330187 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:18.330574 (+ 387us) server_negotiation.cc:299] Negotiation successful
0806 04:51:18.330926 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":339,"threads_started":1}
I20260806 04:51:18.338652 14526 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea823db379f9475fa5da383cb5036182" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41181 } }
I20260806 04:51:18.338989 14526 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:18.339056 14526 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea823db379f9475fa5da383cb5036182, State: Initialized, Role: FOLLOWER
I20260806 04:51:18.339475 14526 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea823db379f9475fa5da383cb5036182" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41181 } }
I20260806 04:51:18.339660 14526 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:51:18.339744 14526 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:51:18.339852 14526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:18.341823 14526 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea823db379f9475fa5da383cb5036182" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41181 } }
I20260806 04:51:18.342233 14526 leader_election.cc:302] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea823db379f9475fa5da383cb5036182; no voters:
I20260806 04:51:18.342660 14526 leader_election.cc:288] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:51:18.342792 14531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:18.343814 14531 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [term 1 LEADER]: Becoming Leader. State: Replica: ea823db379f9475fa5da383cb5036182, State: Running, Role: LEADER
I20260806 04:51:18.344345 14531 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea823db379f9475fa5da383cb5036182" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41181 } }
I20260806 04:51:18.344604 14526 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:51:18.347283 14533 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea823db379f9475fa5da383cb5036182. Latest consensus state: current_term: 1 leader_uuid: "ea823db379f9475fa5da383cb5036182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea823db379f9475fa5da383cb5036182" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41181 } } }
I20260806 04:51:18.347817 14533 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:18.348265 14532 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea823db379f9475fa5da383cb5036182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea823db379f9475fa5da383cb5036182" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41181 } } }
I20260806 04:51:18.348646 14532 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:18.349510 14539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:51:18.354314 14539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:51:18.362540 14539 catalog_manager.cc:1347] Generated new cluster ID: 146ea8c87b2d426cb23740bc6e21ce5d
I20260806 04:51:18.362628 14539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:51:18.376063 14539 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:51:18.377288 14539 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:51:18.391949 14539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea823db379f9475fa5da383cb5036182: Generated new TSK 0
I20260806 04:51:18.392831 14539 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:51:18.446228 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41181
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:18.752846 14553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:18.753072 14553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:18.753151 14553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:18.759389 14553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:18.759471 14553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:18.759637 14553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:51:18.767590 14553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41181
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:18.768605 14553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:18.769750 14553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:18.781481 14558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:18.781450 14561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:18.781503 14559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:18.782147 14553 server_base.cc:1034] running on GCE node
I20260806 04:51:18.782629 14553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:18.783399 14553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:18.784583 14553 hybrid_clock.cc:648] HybridClock initialized: now 1785991878784551 us; error 27 us; skew 500 ppm
I20260806 04:51:18.784921 14553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:18 dist-test-slave-1x4g krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991878, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:18.787791 14553 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:51:18.789090 14553 webserver.cc:460] Webserver started at http://127.13.148.129:33155/ using document root <none> and password file <none>
I20260806 04:51:18.789654 14553 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:18.789781 14553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:18.790057 14553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:18.792543 14553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "bb96fb29ec674c7e97de25f2c1f44542"
format_stamp: "Formatted at 2026-08-06 04:51:18 on dist-test-slave-1x4g"
server_key: "92e2ec56b13acdbfc0c7c9197938d880"
server_key_iv: "207e85f3ff5e7e91bdb1add72b3bbad9"
server_key_version: "encryptionkey@0"
I20260806 04:51:18.793255 14553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "bb96fb29ec674c7e97de25f2c1f44542"
format_stamp: "Formatted at 2026-08-06 04:51:18 on dist-test-slave-1x4g"
server_key: "92e2ec56b13acdbfc0c7c9197938d880"
server_key_iv: "207e85f3ff5e7e91bdb1add72b3bbad9"
server_key_version: "encryptionkey@0"
I20260806 04:51:18.797909 14553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:51:18.801493 14569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:18.802613 14553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260806 04:51:18.802780 14553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "bb96fb29ec674c7e97de25f2c1f44542"
format_stamp: "Formatted at 2026-08-06 04:51:18 on dist-test-slave-1x4g"
server_key: "92e2ec56b13acdbfc0c7c9197938d880"
server_key_iv: "207e85f3ff5e7e91bdb1add72b3bbad9"
server_key_version: "encryptionkey@0"
I20260806 04:51:18.802982 14553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:18.824153 14553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:18.828120 14553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:18.828414 14553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:18.829624 14553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:18.831387 14553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:18.831478 14553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:18.831626 14553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:18.831691 14553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:18.863286 14553 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:42095
I20260806 04:51:18.863366 14683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:42095 every 8 connection(s)
I20260806 04:51:18.864523 14553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:51:18.871289 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 14553
I20260806 04:51:18.871479 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:51:18.872038 13906 external_mini_cluster.cc:1428] Setting key b8c8c67c9b10e795eaede3335312f2aa
Aug 06 04:51:18 dist-test-slave-1x4g krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991878, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:18.882910 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:18.866949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:48637 (local address 127.13.148.190:41181)
0806 04:51:18.867377 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:18.867382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:18.868259 (+ 877us) server_negotiation.cc:407] Connection header received
0806 04:51:18.870106 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:18.870111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:18.870193 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:18.870301 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:18.871731 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:18.872423 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:18.874159 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:18.874531 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:18.874652 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:18.877845 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:18.877850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:18.877853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:18.877883 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:18.880161 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:18.880839 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:18.880849 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:18.880851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:18.880980 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:18.881393 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:18.881398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:18.881401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:18.881656 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:18.881816 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:18.882398 (+ 582us) server_negotiation.cc:299] Negotiation successful
0806 04:51:18.882589 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":231,"threads_started":1}
I20260806 04:51:18.883723 14688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:18.867384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41181 (local address 127.13.148.129:48637)
0806 04:51:18.868095 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:18.868130 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:18.869834 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:18.870519 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:18.870527 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:18.870988 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:18.871566 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:18.871593 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:18.872589 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:18.872594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:18.874021 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:18.874032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:18.874730 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:18.874736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:18.874930 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:18.875557 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:18.875575 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:18.877630 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:18.880288 (+ 2658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:18.880295 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:18.880308 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:18.880710 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:18.881097 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:18.881101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:18.881104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:18.881270 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:18.881795 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:18.881802 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:18.882095 (+ 293us) client_negotiation.cc:769] Sending connection context
0806 04:51:18.882315 (+ 220us) client_negotiation.cc:240] Negotiation successful
0806 04:51:18.882753 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":254,"threads_started":1}
I20260806 04:51:18.885224 14684 heartbeater.cc:347] Connected to a master server at 127.13.148.190:41181
I20260806 04:51:18.885543 14684 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:18.886359 14684 heartbeater.cc:511] Master 127.13.148.190:41181 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
I20260806 04:51:18.888764 14491 ts_manager.cc:194] Registered new tserver with Master: bb96fb29ec674c7e97de25f2c1f44542 (127.13.148.129:42095)
I20260806 04:51:18.890173 14491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:48637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:51:18.909665 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41181
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:19.198910 14691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:19.199154 14691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:19.199254 14691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:19.205559 14691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:19.205669 14691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:19.205832 14691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:51:19.214413 14691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41181
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:19.215613 14691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:19.217025 14691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:19.228993 14696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:19.229017 14697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:19.229072 14699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:19.229888 14691 server_base.cc:1034] running on GCE node
I20260806 04:51:19.230413 14691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:19.231235 14691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:19.232445 14691 hybrid_clock.cc:648] HybridClock initialized: now 1785991879232406 us; error 57 us; skew 500 ppm
I20260806 04:51:19.232717 14691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:19 dist-test-slave-1x4g krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991879, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:19.235421 14691 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:51:19.236558 14691 webserver.cc:460] Webserver started at http://127.13.148.130:38847/ using document root <none> and password file <none>
I20260806 04:51:19.237291 14691 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:19.237435 14691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:19.237736 14691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:19.240386 14691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "6cf9a9f973524bcb99d16c3728064ff8"
format_stamp: "Formatted at 2026-08-06 04:51:19 on dist-test-slave-1x4g"
server_key: "cf652a1a810c6c4b713858588b658b60"
server_key_iv: "034f06c5b886e942ec63637c252ea81a"
server_key_version: "encryptionkey@0"
I20260806 04:51:19.241153 14691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "6cf9a9f973524bcb99d16c3728064ff8"
format_stamp: "Formatted at 2026-08-06 04:51:19 on dist-test-slave-1x4g"
server_key: "cf652a1a810c6c4b713858588b658b60"
server_key_iv: "034f06c5b886e942ec63637c252ea81a"
server_key_version: "encryptionkey@0"
I20260806 04:51:19.245836 14691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:51:19.249136 14706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:19.250391 14691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 04:51:19.250545 14691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "6cf9a9f973524bcb99d16c3728064ff8"
format_stamp: "Formatted at 2026-08-06 04:51:19 on dist-test-slave-1x4g"
server_key: "cf652a1a810c6c4b713858588b658b60"
server_key_iv: "034f06c5b886e942ec63637c252ea81a"
server_key_version: "encryptionkey@0"
I20260806 04:51:19.250715 14691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:19.265226 14691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:19.269238 14691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:19.269517 14691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:19.270373 14691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:19.271478 14691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:19.271586 14691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:19.271728 14691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:19.271792 14691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:19.301748 14691 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:45901
I20260806 04:51:19.301795 14820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:45901 every 8 connection(s)
I20260806 04:51:19.302934 14691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:51:19.303354 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 14691
I20260806 04:51:19.303550 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:51:19.304145 13906 external_mini_cluster.cc:1428] Setting key e54f0030ab2646615b127272a14fa14a
Aug 06 04:51:19 dist-test-slave-1x4g krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991879, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:19.325984 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:19.305639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:47669 (local address 127.13.148.190:41181)
0806 04:51:19.305891 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:19.305897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:19.307735 (+ 1838us) server_negotiation.cc:407] Connection header received
0806 04:51:19.310535 (+ 2800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:19.310541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:19.310619 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:19.310719 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:19.312567 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.313378 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.314830 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.315241 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.315364 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:19.319490 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:19.319517 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:19.319523 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:19.319562 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:19.323018 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:19.323880 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:19.323883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:19.323885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:19.324002 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:19.324431 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:19.324434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:19.324436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:19.324646 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:19.324746 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:19.325602 (+ 856us) server_negotiation.cc:299] Negotiation successful
0806 04:51:19.325758 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
I20260806 04:51:19.327363 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:19.306170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41181 (local address 127.13.148.130:47669)
0806 04:51:19.307565 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:19.307621 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:19.310243 (+ 2622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:19.310981 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:19.310994 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:19.311774 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:19.312401 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:19.312418 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.313544 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.313549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:19.314691 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:19.314703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.315447 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.315464 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:19.315682 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:19.316337 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:19.316368 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:19.319294 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:19.323146 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:19.323157 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:19.323173 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:19.323776 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:19.324106 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:19.324110 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:19.324113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:19.324328 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:19.324767 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:19.324787 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:19.325206 (+ 419us) client_negotiation.cc:769] Sending connection context
0806 04:51:19.325517 (+ 311us) client_negotiation.cc:240] Negotiation successful
0806 04:51:19.326071 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1088,"thread_start_us":883,"threads_started":1}
I20260806 04:51:19.328747 14821 heartbeater.cc:347] Connected to a master server at 127.13.148.190:41181
I20260806 04:51:19.329077 14821 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
I20260806 04:51:19.329865 14821 heartbeater.cc:511] Master 127.13.148.190:41181 requested a full tablet report, sending...
I20260806 04:51:19.331318 14491 ts_manager.cc:194] Registered new tserver with Master: 6cf9a9f973524bcb99d16c3728064ff8 (127.13.148.130:45901)
I20260806 04:51:19.332425 14491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:47669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
I20260806 04:51:19.349341 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41181
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:19.645309 14826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:19.645543 14826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:19.645625 14826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:19.651882 14826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:19.651985 14826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:19.652133 14826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:51:19.661379 14826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41181
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:19.662545 14826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:19.663723 14826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:19.675266 14832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:19.675196 14831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:19.675583 14834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:19.676343 14826 server_base.cc:1034] running on GCE node
I20260806 04:51:19.676923 14826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:19.677726 14826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:19.678928 14826 hybrid_clock.cc:648] HybridClock initialized: now 1785991879678895 us; error 24 us; skew 500 ppm
I20260806 04:51:19.679178 14826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:19 dist-test-slave-1x4g krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991879, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:19.682003 14826 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:51:19.683261 14826 webserver.cc:460] Webserver started at http://127.13.148.131:38461/ using document root <none> and password file <none>
I20260806 04:51:19.683840 14826 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:19.683957 14826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:19.684185 14826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:19.686630 14826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "e77c5cf1b42b4e19af577419b0fbaabb"
format_stamp: "Formatted at 2026-08-06 04:51:19 on dist-test-slave-1x4g"
server_key: "4febd45725d793614c40e2cc1d3d0797"
server_key_iv: "d2fec1e5de52028543c00c29f6f650b9"
server_key_version: "encryptionkey@0"
I20260806 04:51:19.687341 14826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "e77c5cf1b42b4e19af577419b0fbaabb"
format_stamp: "Formatted at 2026-08-06 04:51:19 on dist-test-slave-1x4g"
server_key: "4febd45725d793614c40e2cc1d3d0797"
server_key_iv: "d2fec1e5de52028543c00c29f6f650b9"
server_key_version: "encryptionkey@0"
I20260806 04:51:19.691800 14826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:51:19.694751 14841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:19.695812 14826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:51:19.695926 14826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "e77c5cf1b42b4e19af577419b0fbaabb"
format_stamp: "Formatted at 2026-08-06 04:51:19 on dist-test-slave-1x4g"
server_key: "4febd45725d793614c40e2cc1d3d0797"
server_key_iv: "d2fec1e5de52028543c00c29f6f650b9"
server_key_version: "encryptionkey@0"
I20260806 04:51:19.696067 14826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:19.710597 14826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:19.714128 14826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:19.714321 14826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:19.715042 14826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:19.716111 14826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:19.716176 14826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:19.716251 14826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:19.716290 14826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:19.747309 14826 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:35157
I20260806 04:51:19.747390 14958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:35157 every 8 connection(s)
I20260806 04:51:19.748556 14826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:51:19.753646 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 14826
I20260806 04:51:19.753841 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:51:19.754271 13906 external_mini_cluster.cc:1428] Setting key 65c1fe7d0ffdb94b666ac8e637172dbd
Aug 06 04:51:19 dist-test-slave-1x4g krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991879, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:19.766673 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:19.750967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:56955 (local address 127.13.148.190:41181)
0806 04:51:19.751160 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:19.751164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:19.752455 (+ 1291us) server_negotiation.cc:407] Connection header received
0806 04:51:19.754482 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:19.754488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:19.754576 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:19.754676 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:19.756095 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.756914 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.758413 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.758786 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.758919 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:19.761863 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:19.761867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:19.761870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:19.761899 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:19.764020 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:19.764617 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:19.764623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:19.764626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:19.764747 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:19.765184 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:19.765188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:19.765191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:19.765431 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:19.765584 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:19.766166 (+ 582us) server_negotiation.cc:299] Negotiation successful
0806 04:51:19.766362 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:51:19.767704 14961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:19.751499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41181 (local address 127.13.148.131:56955)
0806 04:51:19.752290 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:19.752327 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:19.754192 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:19.754925 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:19.754933 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:19.755403 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:19.755928 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:19.755943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.757076 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.757080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:19.758270 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:19.758281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.759011 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.759019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:19.759222 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:19.759770 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:19.759789 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:19.761667 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:19.764138 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:19.764145 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:19.764157 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:19.764513 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:19.764905 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:19.764908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:19.764912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:19.765082 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:19.765564 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:19.765571 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:19.765841 (+ 270us) client_negotiation.cc:769] Sending connection context
0806 04:51:19.766068 (+ 227us) client_negotiation.cc:240] Negotiation successful
0806 04:51:19.766519 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":383,"threads_started":1}
I20260806 04:51:19.769153 14959 heartbeater.cc:347] Connected to a master server at 127.13.148.190:41181
I20260806 04:51:19.769423 14959 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:19.770026 14959 heartbeater.cc:511] Master 127.13.148.190:41181 requested a full tablet report, sending...
I20260806 04:51:19.771234 14490 ts_manager.cc:194] Registered new tserver with Master: e77c5cf1b42b4e19af577419b0fbaabb (127.13.148.131:35157)
I20260806 04:51:19.771922 14490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:56955
I20260806 04:51:19.781055 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:51:19.799997 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:19.789209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.13.148.190:41181)
0806 04:51:19.789382 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:19.789389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:19.789834 (+ 445us) server_negotiation.cc:407] Connection header received
0806 04:51:19.789945 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:19.789948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:19.790003 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:19.790074 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:19.791387 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.792058 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.793206 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.793549 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.793630 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:19.795458 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:19.795462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:19.795465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:19.795489 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:19.797771 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:19.798325 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:19.798328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:19.798330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:19.798462 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:19.798738 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:19.798741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:19.798742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:19.799010 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:19.799215 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:19.799506 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 04:51:19.799709 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:51:19.807400 14490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:51:19.820900 14490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:51:19.852016 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:19.839637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:45901 (local address 127.0.0.1:37756)
0806 04:51:19.840764 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:19.840825 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:19.841019 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:19.842486 (+ 1467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:19.842497 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:19.842548 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:19.843052 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:19.843067 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.845766 (+ 2699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.845774 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:19.849277 (+ 3503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:19.849296 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.851072 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.851077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:19.851268 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:19.851321 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 04:51:19.851442 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 04:51:19.851608 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":2,"thread_start_us":726,"threads_started":1}
I20260806 04:51:19.852029 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:19.840026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37756 (local address 127.13.148.130:45901)
0806 04:51:19.841791 (+ 1765us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:19.841799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:19.841826 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:51:19.841943 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:19.841951 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:19.842192 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:19.843289 (+ 1097us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:19.843550 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.845600 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.849512 (+ 3912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.850823 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.851059 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:19.851230 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:19.851609 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 04:51:19.851762 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1622,"thread_start_us":1468,"threads_started":1}
I20260806 04:51:19.852281 14971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:19.839665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47530 (local address 127.13.148.129:42095)
0806 04:51:19.840241 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:19.840249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:19.842121 (+ 1872us) server_negotiation.cc:407] Connection header received
0806 04:51:19.842459 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:19.842469 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:19.842704 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:19.842916 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:19.844070 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.845908 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.849889 (+ 3981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.851196 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.851329 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:19.851498 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:19.851800 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 04:51:19.851952 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":309,"threads_started":1}
I20260806 04:51:19.853103 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:19.839638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:42095 (local address 127.0.0.1:47530)
0806 04:51:19.841992 (+ 2354us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:19.842006 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:19.842243 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:19.843277 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:19.843284 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:19.843315 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:19.843835 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:19.843848 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.846146 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.846153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:19.849729 (+ 3576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:19.849747 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.851424 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.851433 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:19.851596 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:19.851622 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:51:19.852611 (+ 989us) client_negotiation.cc:240] Negotiation successful
0806 04:51:19.852711 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2223,"mutex_wait_us":1238,"thread_start_us":1563,"threads_started":1}
I20260806 04:51:19.856133 14756 tablet_service.cc:1467] Processing CreateTablet for tablet 1397883531334e5c8f40068519dfd161 (DEFAULT_TABLE table=test-table [id=61853db886994359adc568e7d64c831e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:19.856431 14619 tablet_service.cc:1467] Processing CreateTablet for tablet 1397883531334e5c8f40068519dfd161 (DEFAULT_TABLE table=test-table [id=61853db886994359adc568e7d64c831e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:19.857527 14756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1397883531334e5c8f40068519dfd161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:19.857698 14975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:19.842098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:35157 (local address 127.0.0.1:36108)
0806 04:51:19.849701 (+ 7603us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:19.849720 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:19.849830 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:19.850430 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:19.850434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:19.850459 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:19.850817 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:19.850827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.852671 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.852675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:19.855299 (+ 2624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:19.855313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.856933 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.856947 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:19.857136 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:19.857168 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 04:51:19.857264 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 04:51:19.857383 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7498,"thread_start_us":7381,"threads_started":1}
I20260806 04:51:19.857844 14619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1397883531334e5c8f40068519dfd161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:19.857831 14976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:19.842493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36108 (local address 127.13.148.131:35157)
0806 04:51:19.849038 (+ 6545us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:19.849045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:19.849823 (+ 778us) server_negotiation.cc:407] Connection header received
0806 04:51:19.849962 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:19.849969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:19.850251 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:19.850422 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:19.851007 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.852515 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.855464 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.856711 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.856940 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:19.857093 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:19.857329 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 04:51:19.857505 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6393,"thread_start_us":6561,"threads_started":1}
I20260806 04:51:19.862592 14892 tablet_service.cc:1467] Processing CreateTablet for tablet 1397883531334e5c8f40068519dfd161 (DEFAULT_TABLE table=test-table [id=61853db886994359adc568e7d64c831e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:19.863855 14892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1397883531334e5c8f40068519dfd161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:19.875833 14977 tablet_bootstrap.cc:492] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8: Bootstrap starting.
I20260806 04:51:19.876818 14978 tablet_bootstrap.cc:492] T 1397883531334e5c8f40068519dfd161 P bb96fb29ec674c7e97de25f2c1f44542: Bootstrap starting.
I20260806 04:51:19.879208 14979 tablet_bootstrap.cc:492] T 1397883531334e5c8f40068519dfd161 P e77c5cf1b42b4e19af577419b0fbaabb: Bootstrap starting.
I20260806 04:51:19.879890 14977 tablet_bootstrap.cc:654] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:19.881246 14977 log.cc:824] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:19.881312 14978 tablet_bootstrap.cc:654] T 1397883531334e5c8f40068519dfd161 P bb96fb29ec674c7e97de25f2c1f44542: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:19.881819 14979 tablet_bootstrap.cc:654] T 1397883531334e5c8f40068519dfd161 P e77c5cf1b42b4e19af577419b0fbaabb: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:19.882434 14978 log.cc:824] T 1397883531334e5c8f40068519dfd161 P bb96fb29ec674c7e97de25f2c1f44542: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:19.883037 14979 log.cc:824] T 1397883531334e5c8f40068519dfd161 P e77c5cf1b42b4e19af577419b0fbaabb: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:19.884369 14977 tablet_bootstrap.cc:492] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8: No bootstrap required, opened a new log
I20260806 04:51:19.884626 14977 ts_tablet_manager.cc:1397] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 04:51:19.885201 14978 tablet_bootstrap.cc:492] T 1397883531334e5c8f40068519dfd161 P bb96fb29ec674c7e97de25f2c1f44542: No bootstrap required, opened a new log
I20260806 04:51:19.885470 14978 ts_tablet_manager.cc:1397] T 1397883531334e5c8f40068519dfd161 P bb96fb29ec674c7e97de25f2c1f44542: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 04:51:19.886684 14979 tablet_bootstrap.cc:492] T 1397883531334e5c8f40068519dfd161 P e77c5cf1b42b4e19af577419b0fbaabb: No bootstrap required, opened a new log
I20260806 04:51:19.886961 14979 ts_tablet_manager.cc:1397] T 1397883531334e5c8f40068519dfd161 P e77c5cf1b42b4e19af577419b0fbaabb: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 04:51:19.893450 14684 heartbeater.cc:503] Master 127.13.148.190:41181 was elected leader, sending a full tablet report...
I20260806 04:51:19.893200 14979 raft_consensus.cc:348] T 1397883531334e5c8f40068519dfd161 P e77c5cf1b42b4e19af577419b0fbaabb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96fb29ec674c7e97de25f2c1f44542" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42095 } } peers { permanent_uuid: "6cf9a9f973524bcb99d16c3728064ff8" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 45901 } } peers { permanent_uuid: "e77c5cf1b42b4e19af577419b0fbaabb" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35157 } }
I20260806 04:51:19.893636 14979 raft_consensus.cc:374] T 1397883531334e5c8f40068519dfd161 P e77c5cf1b42b4e19af577419b0fbaabb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:19.893733 14979 raft_consensus.cc:729] T 1397883531334e5c8f40068519dfd161 P e77c5cf1b42b4e19af577419b0fbaabb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77c5cf1b42b4e19af577419b0fbaabb, State: Initialized, Role: FOLLOWER
I20260806 04:51:19.894274 14979 consensus_queue.cc:260] T 1397883531334e5c8f40068519dfd161 P e77c5cf1b42b4e19af577419b0fbaabb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96fb29ec674c7e97de25f2c1f44542" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42095 } } peers { permanent_uuid: "6cf9a9f973524bcb99d16c3728064ff8" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 45901 } } peers { permanent_uuid: "e77c5cf1b42b4e19af577419b0fbaabb" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35157 } }
I20260806 04:51:19.895439 14977 raft_consensus.cc:348] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96fb29ec674c7e97de25f2c1f44542" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42095 } } peers { permanent_uuid: "6cf9a9f973524bcb99d16c3728064ff8" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 45901 } } peers { permanent_uuid: "e77c5cf1b42b4e19af577419b0fbaabb" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35157 } }
I20260806 04:51:19.895458 14978 raft_consensus.cc:348] T 1397883531334e5c8f40068519dfd161 P bb96fb29ec674c7e97de25f2c1f44542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96fb29ec674c7e97de25f2c1f44542" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42095 } } peers { permanent_uuid: "6cf9a9f973524bcb99d16c3728064ff8" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 45901 } } peers { permanent_uuid: "e77c5cf1b42b4e19af577419b0fbaabb" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35157 } }
I20260806 04:51:19.895792 14977 raft_consensus.cc:374] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:19.895792 14978 raft_consensus.cc:374] T 1397883531334e5c8f40068519dfd161 P bb96fb29ec674c7e97de25f2c1f44542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:19.895877 14977 raft_consensus.cc:729] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf9a9f973524bcb99d16c3728064ff8, State: Initialized, Role: FOLLOWER
I20260806 04:51:19.895874 14978 raft_consensus.cc:729] T 1397883531334e5c8f40068519dfd161 P bb96fb29ec674c7e97de25f2c1f44542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb96fb29ec674c7e97de25f2c1f44542, State: Initialized, Role: FOLLOWER
I20260806 04:51:19.896456 14978 consensus_queue.cc:260] T 1397883531334e5c8f40068519dfd161 P bb96fb29ec674c7e97de25f2c1f44542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96fb29ec674c7e97de25f2c1f44542" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42095 } } peers { permanent_uuid: "6cf9a9f973524bcb99d16c3728064ff8" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 45901 } } peers { permanent_uuid: "e77c5cf1b42b4e19af577419b0fbaabb" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35157 } }
I20260806 04:51:19.896445 14977 consensus_queue.cc:260] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96fb29ec674c7e97de25f2c1f44542" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42095 } } peers { permanent_uuid: "6cf9a9f973524bcb99d16c3728064ff8" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 45901 } } peers { permanent_uuid: "e77c5cf1b42b4e19af577419b0fbaabb" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35157 } }
I20260806 04:51:19.904767 14977 ts_tablet_manager.cc:1428] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260806 04:51:19.904735 14959 heartbeater.cc:503] Master 127.13.148.190:41181 was elected leader, sending a full tablet report...
I20260806 04:51:19.905350 14979 ts_tablet_manager.cc:1428] T 1397883531334e5c8f40068519dfd161 P e77c5cf1b42b4e19af577419b0fbaabb: Time spent starting tablet: real 0.018s user 0.010s sys 0.009s
I20260806 04:51:19.906239 14978 ts_tablet_manager.cc:1428] T 1397883531334e5c8f40068519dfd161 P bb96fb29ec674c7e97de25f2c1f44542: Time spent starting tablet: real 0.021s user 0.020s sys 0.000s
I20260806 04:51:19.906291 14821 heartbeater.cc:503] Master 127.13.148.190:41181 was elected leader, sending a full tablet report...
I20260806 04:51:19.928646 14985 raft_consensus.cc:482] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:51:19.928941 14985 raft_consensus.cc:504] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96fb29ec674c7e97de25f2c1f44542" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42095 } } peers { permanent_uuid: "6cf9a9f973524bcb99d16c3728064ff8" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 45901 } } peers { permanent_uuid: "e77c5cf1b42b4e19af577419b0fbaabb" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35157 } }
I20260806 04:51:19.930307 14985 leader_election.cc:288] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb96fb29ec674c7e97de25f2c1f44542 (127.13.148.129:42095), e77c5cf1b42b4e19af577419b0fbaabb (127.13.148.131:35157)
I20260806 04:51:19.936725 14971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:19.930621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:42913 (local address 127.13.148.129:42095)
0806 04:51:19.930795 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:19.930798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:19.931404 (+ 606us) server_negotiation.cc:407] Connection header received
0806 04:51:19.931759 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:19.931765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:19.931838 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:19.931936 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:19.932567 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.933378 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.935091 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.935843 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.935944 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:19.936024 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:19.936347 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 04:51:19.936464 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:51:19.936764 14976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:19.931440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:33275 (local address 127.13.148.131:35157)
0806 04:51:19.931624 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:19.931630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:19.932057 (+ 427us) server_negotiation.cc:407] Connection header received
0806 04:51:19.932212 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:19.932217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:19.932289 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:19.932384 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:19.932954 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.933719 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.935225 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.935930 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.936075 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:19.936138 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:19.936411 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 04:51:19.936527 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:51:19.936870 14987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:19.931337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:35157 (local address 127.13.148.130:33275)
0806 04:51:19.931952 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:19.931973 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:19.932085 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:19.932400 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:19.932403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:19.932420 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:19.932818 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:19.932832 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.933851 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.933855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:19.935094 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:19.935106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.936133 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.936137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:19.936248 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:19.936268 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 04:51:19.936366 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 04:51:19.936474 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":295,"threads_started":1}
I20260806 04:51:19.936878 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:19.930627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:42095 (local address 127.13.148.130:42913)
0806 04:51:19.931283 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:19.931299 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:19.931535 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:19.931982 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:19.931985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:19.932005 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:19.932410 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:19.932424 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.933510 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.933515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:19.934945 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:19.934957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.936049 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.936054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:19.936210 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:19.936230 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 04:51:19.936294 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 04:51:19.936413 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":1,"thread_start_us":364,"threads_started":1}
I20260806 04:51:19.937916 14639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1397883531334e5c8f40068519dfd161" candidate_uuid: "6cf9a9f973524bcb99d16c3728064ff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb96fb29ec674c7e97de25f2c1f44542" is_pre_election: true
I20260806 04:51:19.937943 14912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1397883531334e5c8f40068519dfd161" candidate_uuid: "6cf9a9f973524bcb99d16c3728064ff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77c5cf1b42b4e19af577419b0fbaabb" is_pre_election: true
I20260806 04:51:19.938484 14639 raft_consensus.cc:2454] T 1397883531334e5c8f40068519dfd161 P bb96fb29ec674c7e97de25f2c1f44542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cf9a9f973524bcb99d16c3728064ff8 in term 0.
I20260806 04:51:19.938476 14912 raft_consensus.cc:2454] T 1397883531334e5c8f40068519dfd161 P e77c5cf1b42b4e19af577419b0fbaabb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cf9a9f973524bcb99d16c3728064ff8 in term 0.
I20260806 04:51:19.939116 14710 leader_election.cc:302] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf9a9f973524bcb99d16c3728064ff8, e77c5cf1b42b4e19af577419b0fbaabb; no voters:
I20260806 04:51:19.939412 14985 raft_consensus.cc:2781] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:51:19.939535 14985 raft_consensus.cc:482] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:51:19.939661 14985 raft_consensus.cc:3037] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:19.941656 14985 raft_consensus.cc:504] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96fb29ec674c7e97de25f2c1f44542" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42095 } } peers { permanent_uuid: "6cf9a9f973524bcb99d16c3728064ff8" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 45901 } } peers { permanent_uuid: "e77c5cf1b42b4e19af577419b0fbaabb" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35157 } }
I20260806 04:51:19.942160 14985 leader_election.cc:288] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 [CANDIDATE]: Term 1 election: Requested vote from peers bb96fb29ec674c7e97de25f2c1f44542 (127.13.148.129:42095), e77c5cf1b42b4e19af577419b0fbaabb (127.13.148.131:35157)
I20260806 04:51:19.942551 14639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1397883531334e5c8f40068519dfd161" candidate_uuid: "6cf9a9f973524bcb99d16c3728064ff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb96fb29ec674c7e97de25f2c1f44542"
I20260806 04:51:19.942689 14912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1397883531334e5c8f40068519dfd161" candidate_uuid: "6cf9a9f973524bcb99d16c3728064ff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77c5cf1b42b4e19af577419b0fbaabb"
I20260806 04:51:19.942727 14639 raft_consensus.cc:3037] T 1397883531334e5c8f40068519dfd161 P bb96fb29ec674c7e97de25f2c1f44542 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:19.942873 14912 raft_consensus.cc:3037] T 1397883531334e5c8f40068519dfd161 P e77c5cf1b42b4e19af577419b0fbaabb [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:19.944983 14639 raft_consensus.cc:2454] T 1397883531334e5c8f40068519dfd161 P bb96fb29ec674c7e97de25f2c1f44542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cf9a9f973524bcb99d16c3728064ff8 in term 1.
I20260806 04:51:19.945096 14912 raft_consensus.cc:2454] T 1397883531334e5c8f40068519dfd161 P e77c5cf1b42b4e19af577419b0fbaabb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cf9a9f973524bcb99d16c3728064ff8 in term 1.
I20260806 04:51:19.945367 14710 leader_election.cc:302] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf9a9f973524bcb99d16c3728064ff8, bb96fb29ec674c7e97de25f2c1f44542; no voters:
I20260806 04:51:19.945633 14985 raft_consensus.cc:2781] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:19.946028 14985 raft_consensus.cc:686] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 [term 1 LEADER]: Becoming Leader. State: Replica: 6cf9a9f973524bcb99d16c3728064ff8, State: Running, Role: LEADER
I20260806 04:51:19.946499 14985 consensus_queue.cc:237] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96fb29ec674c7e97de25f2c1f44542" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42095 } } peers { permanent_uuid: "6cf9a9f973524bcb99d16c3728064ff8" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 45901 } } peers { permanent_uuid: "e77c5cf1b42b4e19af577419b0fbaabb" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35157 } }
I20260806 04:51:19.953586 14490 catalog_manager.cc:5495] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cf9a9f973524bcb99d16c3728064ff8 (127.13.148.130). New cstate: current_term: 1 leader_uuid: "6cf9a9f973524bcb99d16c3728064ff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96fb29ec674c7e97de25f2c1f44542" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cf9a9f973524bcb99d16c3728064ff8" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 45901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77c5cf1b42b4e19af577419b0fbaabb" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35157 } health_report { overall_health: UNKNOWN } } }
Aug 06 04:51:19 dist-test-slave-1x4g krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:51:19.978444 14971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:19.964387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47542 (local address 127.13.148.129:42095)
0806 04:51:19.964527 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:19.964530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:19.965061 (+ 531us) server_negotiation.cc:407] Connection header received
0806 04:51:19.965232 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:19.965237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:19.965298 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:19.965388 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:19.966773 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.967530 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.968790 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:19.969169 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:19.969289 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:19.973091 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:19.973103 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:19.973116 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:19.973165 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:19.975601 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:19.976275 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:19.976282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:19.976284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:19.976400 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:19.976810 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:19.976816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:19.976819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:19.977218 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:19.977523 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:19.977825 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 04:51:19.978066 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 04:51:20.000208 14960 tablet.cc:2369] T 1397883531334e5c8f40068519dfd161 P e77c5cf1b42b4e19af577419b0fbaabb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 04:51:19 dist-test-slave-1x4g krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991879, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 04:51:20 dist-test-slave-1x4g krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991879, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:51:20.023290 14971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:20.009738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.13.148.129:42095)
0806 04:51:20.009969 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:20.009975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:20.010236 (+ 261us) server_negotiation.cc:407] Connection header received
0806 04:51:20.010518 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:20.010527 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:20.010635 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:20.010752 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:20.012090 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:20.012894 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:20.014140 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:20.014548 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:20.014710 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:20.018718 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:20.018722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:20.018725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:20.018755 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:20.021261 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:20.021816 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:20.021831 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:20.021833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:20.021948 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:20.022228 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:20.022231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:20.022233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:20.022475 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:20.022659 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:20.022862 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 04:51:20.023011 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 04:51:20 dist-test-slave-1x4g krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260806 04:51:20.055035 14822 tablet.cc:2369] T 1397883531334e5c8f40068519dfd161 P 6cf9a9f973524bcb99d16c3728064ff8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 04:51:20 dist-test-slave-1x4g krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:51:20.069846 14971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:20.057050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47552 (local address 127.13.148.129:42095)
0806 04:51:20.057291 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:20.057298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:20.057582 (+ 284us) server_negotiation.cc:407] Connection header received
0806 04:51:20.057796 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:20.057799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:20.057864 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:20.057982 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:20.059285 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:20.059965 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:20.061183 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:20.061594 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:20.061723 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:20.065324 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:20.065329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:20.065332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:20.065367 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:20.067702 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:20.068267 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:20.068271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:20.068273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:20.068390 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:20.068703 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:20.068705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:20.068707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:20.069040 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:20.069209 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:20.069435 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 04:51:20.069596 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260806 04:51:20.071554 14599 tablet_service.cc:2908] Rejecting scan request for tablet 1397883531334e5c8f40068519dfd161: Uninitialized: clean time has not yet been initialized
I20260806 04:51:20.084808 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 14553
I20260806 04:51:20.096477 14971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:20.085781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47564 (local address 127.13.148.129:42095)
0806 04:51:20.085948 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:20.085953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:20.086299 (+ 346us) server_negotiation.cc:407] Connection header received
0806 04:51:20.086417 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:20.086433 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:20.086539 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:20.086613 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:20.087712 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:20.088367 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:20.089491 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:20.089863 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:20.089945 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:20.091787 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:20.091791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:20.091794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:20.091823 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:20.094394 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:20.094947 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:20.094951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:20.094955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:20.095061 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:20.095373 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:20.095376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:20.095378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:20.095651 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:20.095825 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:20.096070 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 04:51:20.096227 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:51:20.097179 14679 generic_service.cc:196] Checking for leaks (request via RPC)
W20260806 04:51:20.180446 14686 tablet.cc:2369] T 1397883531334e5c8f40068519dfd161 P bb96fb29ec674c7e97de25f2c1f44542: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:51:20.181170 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 14553
I20260806 04:51:20.198966 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 14691
Aug 06 04:51:20 dist-test-slave-1x4g krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:51:20.211928 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:20.199698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37772 (local address 127.13.148.130:45901)
0806 04:51:20.199901 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:20.199909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:20.199934 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:51:20.200013 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:20.200017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:20.200088 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:20.200165 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:20.201398 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:20.202090 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:20.203192 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:20.203554 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:20.203632 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:20.207090 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:20.207102 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:20.207118 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:20.207147 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:20.209704 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:20.210227 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:20.210233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:20.210235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:20.210366 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:20.210692 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:20.210695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:20.210697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:20.211109 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:20.211310 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:20.211520 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 04:51:20.211702 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:51:20.212488 14816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:20.292870 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 14691
I20260806 04:51:20.310412 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 14826
Aug 06 04:51:20 dist-test-slave-1x4g krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
I20260806 04:51:20.324404 14976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:20.311120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36116 (local address 127.13.148.131:35157)
0806 04:51:20.311271 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:20.311276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:20.311293 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:51:20.311432 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:20.311437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:20.311506 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:20.311598 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:20.312851 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:20.313920 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:20.315136 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:20.315731 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:20.315844 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:20.319753 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:20.319765 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:20.319780 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:20.319812 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:20.322428 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:20.322945 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:20.322951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:20.322953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:20.323068 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:20.323327 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:20.323330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:20.323331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:20.323638 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:20.323824 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:20.324050 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 04:51:20.324182 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:51:20.325031 14954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:20.403573 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 14826
I20260806 04:51:20.421401 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 14458
I20260806 04:51:20.422580 14521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:20.521608 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 14458
2026-08-06T04:51:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3744 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:51:20 dist-test-slave-1x4g krb5kdc[15007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:51:20 dist-test-slave-1x4g krb5kdc[15007](info): set up 2 sockets
Aug 06 04:51:20 dist-test-slave-1x4g krb5kdc[15007](info): commencing operation
krb5kdc: starting...
W20260806 04:51:21.532425 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:51:21 dist-test-slave-1x4g krb5kdc[15007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:51:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:51:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:51:21.687863 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:44125
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:44571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:44125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:21.978154 15020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:21.978403 15020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:21.978477 15020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:21.984400 15020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:51:21.984509 15020 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 04:51:21.984577 15020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:21.984676 15020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:21.984755 15020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:51:21.984860 15020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:51:21.993306 15020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:44571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:44125
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:44125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:21.994210 15020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:21.995301 15020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:22.006031 15025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:22.006111 15026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:22.006073 15029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:22.007071 15020 server_base.cc:1034] running on GCE node
I20260806 04:51:22.007813 15020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:22.009075 15020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:22.010293 15020 hybrid_clock.cc:648] HybridClock initialized: now 1785991882010229 us; error 46 us; skew 500 ppm
I20260806 04:51:22.010582 15020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:22 dist-test-slave-1x4g krb5kdc[15007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991882, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:22.013505 15020 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:51:22.014799 15020 webserver.cc:460] Webserver started at http://127.13.148.190:39473/ using document root <none> and password file <none>
I20260806 04:51:22.015398 15020 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:22.015547 15020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:22.015846 15020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:22.018726 15020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "f0aa30ca2b61432b9b59686ee5c9f5df"
format_stamp: "Formatted at 2026-08-06 04:51:22 on dist-test-slave-1x4g"
server_key: "b20eeb9cc2ac916f2bf2e67d55f95161"
server_key_iv: "f58a6208c0579bcb8e42775653407a94"
server_key_version: "encryptionkey@0"
I20260806 04:51:22.019515 15020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "f0aa30ca2b61432b9b59686ee5c9f5df"
format_stamp: "Formatted at 2026-08-06 04:51:22 on dist-test-slave-1x4g"
server_key: "b20eeb9cc2ac916f2bf2e67d55f95161"
server_key_iv: "f58a6208c0579bcb8e42775653407a94"
server_key_version: "encryptionkey@0"
I20260806 04:51:22.024307 15020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:51:22.027498 15036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:22.028661 15020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:51:22.028872 15020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "f0aa30ca2b61432b9b59686ee5c9f5df"
format_stamp: "Formatted at 2026-08-06 04:51:22 on dist-test-slave-1x4g"
server_key: "b20eeb9cc2ac916f2bf2e67d55f95161"
server_key_iv: "f58a6208c0579bcb8e42775653407a94"
server_key_version: "encryptionkey@0"
I20260806 04:51:22.029067 15020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:22.046815 15020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:22.050271 15020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:22.050513 15020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:22.071076 15020 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:44125
I20260806 04:51:22.071094 15098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:44125 every 8 connection(s)
I20260806 04:51:22.072460 15020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:51:22.075922 15099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:22.082491 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 15020
I20260806 04:51:22.082703 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:51:22.083294 13906 external_mini_cluster.cc:1428] Setting key 9824c1b6e886bb4501d8cc577fd37b4b
I20260806 04:51:22.085570 15099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df: Bootstrap starting.
I20260806 04:51:22.088681 15099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:22.089993 15099 log.cc:824] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:22.092959 15099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df: No bootstrap required, opened a new log
Aug 06 04:51:22 dist-test-slave-1x4g krb5kdc[15007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:22.101963 15099 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0aa30ca2b61432b9b59686ee5c9f5df" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44125 } }
I20260806 04:51:22.102357 15099 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:22.102506 15099 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0aa30ca2b61432b9b59686ee5c9f5df, State: Initialized, Role: FOLLOWER
I20260806 04:51:22.103164 15099 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0aa30ca2b61432b9b59686ee5c9f5df" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44125 } }
I20260806 04:51:22.103418 15099 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:51:22.103569 15099 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:51:22.103783 15099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:22.105762 15102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:22.085105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52296 (local address 127.13.148.190:44125)
0806 04:51:22.085862 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:22.085872 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:22.085909 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:51:22.087603 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:22.087625 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:22.087963 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:22.088255 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:22.090057 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:22.091360 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:22.093027 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:22.093663 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:22.093797 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:22.098623 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:22.098630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:22.098642 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:22.098676 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:22.101706 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:22.102439 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:22.102446 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:22.102457 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:22.102689 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:22.103094 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:22.103097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:22.103099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:22.103731 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:22.104046 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:22.104430 (+ 384us) server_negotiation.cc:299] Negotiation successful
0806 04:51:22.104785 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":371,"threads_started":1}
I20260806 04:51:22.106288 15099 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0aa30ca2b61432b9b59686ee5c9f5df" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44125 } }
I20260806 04:51:22.106829 15099 leader_election.cc:302] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0aa30ca2b61432b9b59686ee5c9f5df; no voters:
I20260806 04:51:22.107514 15099 leader_election.cc:288] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:51:22.107618 15104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:22.110069 15099 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:51:22.110381 15104 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [term 1 LEADER]: Becoming Leader. State: Replica: f0aa30ca2b61432b9b59686ee5c9f5df, State: Running, Role: LEADER
I20260806 04:51:22.110883 15104 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0aa30ca2b61432b9b59686ee5c9f5df" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44125 } }
I20260806 04:51:22.113746 15105 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0aa30ca2b61432b9b59686ee5c9f5df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0aa30ca2b61432b9b59686ee5c9f5df" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44125 } } }
I20260806 04:51:22.113833 15106 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0aa30ca2b61432b9b59686ee5c9f5df. Latest consensus state: current_term: 1 leader_uuid: "f0aa30ca2b61432b9b59686ee5c9f5df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0aa30ca2b61432b9b59686ee5c9f5df" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44125 } } }
I20260806 04:51:22.114068 15105 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:22.114148 15106 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:22.115437 15112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:51:22.120580 15112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:51:22.129246 15112 catalog_manager.cc:1347] Generated new cluster ID: 03ebe1d94e514a438c7c4a003bf267a9
I20260806 04:51:22.129336 15112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:51:22.147066 15112 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:51:22.148972 15112 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:51:22.156621 15112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df: Generated new TSK 0
I20260806 04:51:22.157624 15112 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:51:22.210489 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44125
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:44571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:22.522552 15125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:22.522787 15125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:22.522876 15125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:22.529387 15125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:22.529492 15125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:22.529628 15125 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:51:22.529740 15125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:51:22.538970 15125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:44571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.190:44125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:22.540117 15125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:22.541368 15125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:22.553309 15130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:22.553287 15131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:22.553742 15133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:22.554394 15125 server_base.cc:1034] running on GCE node
I20260806 04:51:22.554948 15125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:22.555893 15125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:22.557134 15125 hybrid_clock.cc:648] HybridClock initialized: now 1785991882557072 us; error 57 us; skew 500 ppm
I20260806 04:51:22.557412 15125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:22 dist-test-slave-1x4g krb5kdc[15007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991882, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:22.560250 15125 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:51:22.561532 15125 webserver.cc:460] Webserver started at http://127.13.148.129:37763/ using document root <none> and password file <none>
I20260806 04:51:22.562160 15125 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:22.562276 15125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:22.562569 15125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:22.565080 15125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "190658c7116946e3ba221898fdd4d4c8"
format_stamp: "Formatted at 2026-08-06 04:51:22 on dist-test-slave-1x4g"
server_key: "8ef53bb1d9ee7af8ef59f7d7a8e1f708"
server_key_iv: "e9fcd65cb335c709eedbe9ddb93255c4"
server_key_version: "encryptionkey@0"
I20260806 04:51:22.565791 15125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "190658c7116946e3ba221898fdd4d4c8"
format_stamp: "Formatted at 2026-08-06 04:51:22 on dist-test-slave-1x4g"
server_key: "8ef53bb1d9ee7af8ef59f7d7a8e1f708"
server_key_iv: "e9fcd65cb335c709eedbe9ddb93255c4"
server_key_version: "encryptionkey@0"
I20260806 04:51:22.570288 15125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:51:22.573416 15140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:22.574532 15125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:51:22.574704 15125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "190658c7116946e3ba221898fdd4d4c8"
format_stamp: "Formatted at 2026-08-06 04:51:22 on dist-test-slave-1x4g"
server_key: "8ef53bb1d9ee7af8ef59f7d7a8e1f708"
server_key_iv: "e9fcd65cb335c709eedbe9ddb93255c4"
server_key_version: "encryptionkey@0"
I20260806 04:51:22.574906 15125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:22.591022 15125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:22.594854 15125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:22.595090 15125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:22.597509 15125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:22.597613 15125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:22.597765 15125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:22.597867 15125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 06 04:51:22 dist-test-slave-1x4g krb5kdc[15007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991882, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:22.621661 15102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:22.598035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:60317 (local address 127.13.148.190:44125)
0806 04:51:22.598233 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:22.598239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:22.599457 (+ 1218us) server_negotiation.cc:407] Connection header received
0806 04:51:22.601577 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:22.601583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:22.601680 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:22.601786 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:22.604253 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:22.605266 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:22.608208 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:22.608726 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:22.608897 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:22.613890 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:22.613896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:22.613901 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:22.613944 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:22.617227 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:22.618292 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:22.618297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:22.618302 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:22.618486 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:22.619034 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:22.619038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:22.619043 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:22.619396 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:22.619536 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:22.620949 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0806 04:51:22.621216 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 04:51:22.621773 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:22.598529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:44125 (local address 127.13.148.129:60317)
0806 04:51:22.599302 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:22.599350 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:22.601313 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:22.602034 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:22.602045 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:22.603087 (+ 1042us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:22.604032 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:22.604058 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:22.605447 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:22.605454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:22.608016 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:22.608037 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:22.609021 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:22.609031 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:22.609358 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:22.610159 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:22.610195 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:22.613602 (+ 3407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:22.617354 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:22.617362 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:22.617378 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:22.618108 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:22.618615 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:22.618620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:22.618624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:22.618914 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:22.619533 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:22.619540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:22.619991 (+ 451us) client_negotiation.cc:769] Sending connection context
0806 04:51:22.620300 (+ 309us) client_negotiation.cc:240] Negotiation successful
0806 04:51:22.620758 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":333,"threads_started":1}
I20260806 04:51:22.641749 15225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:22.626955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:44125 (local address 127.13.148.129:33519)
0806 04:51:22.627606 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:22.627626 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:22.627724 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:22.628054 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:22.628056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:22.628459 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:22.628698 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:22.628705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:22.630255 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:22.630258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:22.631151 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:22.631161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:22.632208 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:22.632255 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:22.632451 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:22.633110 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:22.633136 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:22.634333 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:22.638483 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:22.638487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:22.638489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:22.639114 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:22.639579 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:22.639591 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:22.639597 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:22.639806 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:22.640325 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:22.640332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:22.640553 (+ 221us) client_negotiation.cc:769] Sending connection context
0806 04:51:22.640732 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 04:51:22.641147 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":365,"threads_started":1}
I20260806 04:51:22.642277 15102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:22.626947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:33519 (local address 127.13.148.190:44125)
0806 04:51:22.627155 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:22.627164 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:22.627701 (+ 537us) server_negotiation.cc:407] Connection header received
0806 04:51:22.627815 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:22.627823 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:22.627915 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:22.628054 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:22.628846 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:22.630126 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:22.631272 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:22.631949 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:22.632084 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:22.634577 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:22.634585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:22.634591 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:22.634634 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:22.638313 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:22.639249 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:22.639257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:22.639262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:22.639439 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:22.639934 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:22.639940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:22.639945 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:22.640197 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:22.640363 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:22.641592 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0806 04:51:22.641841 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:51:22.648545 15125 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:43873
I20260806 04:51:22.648563 15262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:43873 every 8 connection(s)
I20260806 04:51:22.649618 15125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:51:22.651391 15263 heartbeater.cc:347] Connected to a master server at 127.13.148.190:44125
I20260806 04:51:22.651609 15263 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:22.652179 15263 heartbeater.cc:511] Master 127.13.148.190:44125 requested a full tablet report, sending...
I20260806 04:51:22.653890 15053 ts_manager.cc:194] Registered new tserver with Master: 190658c7116946e3ba221898fdd4d4c8 (127.13.148.129:43873)
I20260806 04:51:22.655115 15053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:60317
I20260806 04:51:22.657042 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 15125
I20260806 04:51:22.657279 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:51:22.657887 13906 external_mini_cluster.cc:1428] Setting key a4df119bf3c450d2c573ddfd82cbdd22
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:51:22.693323 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44125
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:44571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:22.994756 15267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:22.995045 15267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:22.995138 15267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:23.001820 15267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:23.001928 15267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:23.002029 15267 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:51:23.002106 15267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:51:23.010895 15267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:44571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.190:44125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:23.012185 15267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:23.013409 15267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:23.026221 15273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:23.026288 15272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:23.026254 15275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:23.027346 15267 server_base.cc:1034] running on GCE node
I20260806 04:51:23.027912 15267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:23.028888 15267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:23.030113 15267 hybrid_clock.cc:648] HybridClock initialized: now 1785991883030034 us; error 52 us; skew 500 ppm
I20260806 04:51:23.030396 15267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:23 dist-test-slave-1x4g krb5kdc[15007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991883, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:23.033625 15267 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:51:23.035125 15267 webserver.cc:460] Webserver started at http://127.13.148.130:35441/ using document root <none> and password file <none>
I20260806 04:51:23.035818 15267 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:23.035961 15267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:23.036338 15267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:23.039426 15267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "709f94a1781b4c7896cbf3e366bc19e1"
format_stamp: "Formatted at 2026-08-06 04:51:23 on dist-test-slave-1x4g"
server_key: "1519d8a44ea18e536560a0d565bbaec2"
server_key_iv: "441bd50c9f31f88c62354de0babbc460"
server_key_version: "encryptionkey@0"
I20260806 04:51:23.040244 15267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "709f94a1781b4c7896cbf3e366bc19e1"
format_stamp: "Formatted at 2026-08-06 04:51:23 on dist-test-slave-1x4g"
server_key: "1519d8a44ea18e536560a0d565bbaec2"
server_key_iv: "441bd50c9f31f88c62354de0babbc460"
server_key_version: "encryptionkey@0"
I20260806 04:51:23.045277 15267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 04:51:23.048652 15282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:23.050060 15267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:51:23.050261 15267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "709f94a1781b4c7896cbf3e366bc19e1"
format_stamp: "Formatted at 2026-08-06 04:51:23 on dist-test-slave-1x4g"
server_key: "1519d8a44ea18e536560a0d565bbaec2"
server_key_iv: "441bd50c9f31f88c62354de0babbc460"
server_key_version: "encryptionkey@0"
I20260806 04:51:23.050472 15267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:23.079270 15267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:23.083112 15267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:23.083364 15267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:23.085909 15267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:23.086015 15267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:23.086162 15267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:23.086238 15267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:51:23 dist-test-slave-1x4g krb5kdc[15007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991883, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:23.107903 15102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.086236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:47393 (local address 127.13.148.190:44125)
0806 04:51:23.086429 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:23.086437 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:23.087925 (+ 1488us) server_negotiation.cc:407] Connection header received
0806 04:51:23.090849 (+ 2924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:23.090857 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:23.090944 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:23.091051 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:23.093291 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.094304 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.096740 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.097207 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.097341 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.102263 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:23.102268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:23.102271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:23.102304 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:23.104988 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:23.105704 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:23.105715 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:23.105719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:23.105833 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:23.106273 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:23.106284 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:23.106286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:23.106554 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:23.106710 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:23.107360 (+ 650us) server_negotiation.cc:299] Negotiation successful
0806 04:51:23.107583 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 04:51:23.108927 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.086843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:44125 (local address 127.13.148.130:47393)
0806 04:51:23.087734 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:23.087783 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:23.090533 (+ 2750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:23.091438 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:23.091462 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:23.092161 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:23.093119 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.093140 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.094464 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.094471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.096546 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.096568 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.097480 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.097489 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.097663 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.098426 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:23.098465 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:23.102007 (+ 3542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:23.105132 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:23.105141 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:23.105156 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:23.105575 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:23.105945 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:23.105950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:23.105954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:23.106140 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:23.106693 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:23.106701 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:23.107008 (+ 307us) client_negotiation.cc:769] Sending connection context
0806 04:51:23.107264 (+ 256us) client_negotiation.cc:240] Negotiation successful
0806 04:51:23.107744 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":349,"threads_started":1}
I20260806 04:51:23.132397 15102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.116294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:39259 (local address 127.13.148.190:44125)
0806 04:51:23.116437 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:23.116441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:23.117460 (+ 1019us) server_negotiation.cc:407] Connection header received
0806 04:51:23.117660 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:23.117669 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:23.117761 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:23.117868 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:23.119303 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.120498 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.123073 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.123471 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.123595 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.125706 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:23.125714 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:23.125718 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:23.125754 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:23.129017 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:23.129999 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:23.130007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:23.130011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:23.130163 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:23.130637 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:23.130643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:23.130648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:23.130968 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:23.131070 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:23.131711 (+ 641us) server_negotiation.cc:299] Negotiation successful
0806 04:51:23.131979 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 04:51:23.132477 15356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.116492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:44125 (local address 127.13.148.130:39259)
0806 04:51:23.117376 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:23.117391 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:23.117528 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:23.117916 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:23.117923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:23.118680 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:23.119128 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.119149 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.120691 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.120698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.122917 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.122935 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.123658 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.123664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.123885 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.124659 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:23.124684 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:23.125442 (+ 758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:23.129177 (+ 3735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:23.129185 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:23.129192 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:23.129859 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:23.130294 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:23.130303 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:23.130309 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:23.130491 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:23.131088 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:23.131096 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:23.131390 (+ 294us) client_negotiation.cc:769] Sending connection context
0806 04:51:23.131575 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 04:51:23.131983 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":613,"threads_started":1}
I20260806 04:51:23.134720 15403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:40121 every 8 connection(s)
I20260806 04:51:23.134718 15267 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:40121
I20260806 04:51:23.136212 15267 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:51:23.137822 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 15267
I20260806 04:51:23.138023 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:51:23.138178 15404 heartbeater.cc:347] Connected to a master server at 127.13.148.190:44125
I20260806 04:51:23.138365 15404 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:23.138568 13906 external_mini_cluster.cc:1428] Setting key 3f33f28e648ba4794f4a8aff4f9184e8
I20260806 04:51:23.138864 15404 heartbeater.cc:511] Master 127.13.148.190:44125 requested a full tablet report, sending...
I20260806 04:51:23.140048 15052 ts_manager.cc:194] Registered new tserver with Master: 709f94a1781b4c7896cbf3e366bc19e1 (127.13.148.130:40121)
I20260806 04:51:23.140908 15052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:47393
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
I20260806 04:51:23.175542 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44125
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:44571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:23.468529 15408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:23.468809 15408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:23.469003 15408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:23.475693 15408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:23.475786 15408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:23.475869 15408 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:51:23.476066 15408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:51:23.484710 15408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:44571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.190:44125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:23.485880 15408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:23.487229 15408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:23.499331 15413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:23.499703 15414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:23.499720 15416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:23.500200 15408 server_base.cc:1034] running on GCE node
I20260806 04:51:23.500675 15408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:23.501488 15408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:23.502717 15408 hybrid_clock.cc:648] HybridClock initialized: now 1785991883502669 us; error 28 us; skew 500 ppm
I20260806 04:51:23.502930 15408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:23 dist-test-slave-1x4g krb5kdc[15007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991883, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:23.510597 15408 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:51:23.512055 15408 webserver.cc:460] Webserver started at http://127.13.148.131:43875/ using document root <none> and password file <none>
I20260806 04:51:23.512733 15408 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:23.512874 15408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:23.513206 15408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:23.515761 15408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "b726e3e7aa6b434da437b7840c219baf"
format_stamp: "Formatted at 2026-08-06 04:51:23 on dist-test-slave-1x4g"
server_key: "7ff3302e4461c9a953fcd5429dcdc358"
server_key_iv: "6fdd479b8c9377fa7c2d5fcbb764632f"
server_key_version: "encryptionkey@0"
I20260806 04:51:23.516494 15408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "b726e3e7aa6b434da437b7840c219baf"
format_stamp: "Formatted at 2026-08-06 04:51:23 on dist-test-slave-1x4g"
server_key: "7ff3302e4461c9a953fcd5429dcdc358"
server_key_iv: "6fdd479b8c9377fa7c2d5fcbb764632f"
server_key_version: "encryptionkey@0"
I20260806 04:51:23.521279 15408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260806 04:51:23.524299 15423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:23.525555 15408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:51:23.525712 15408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "b726e3e7aa6b434da437b7840c219baf"
format_stamp: "Formatted at 2026-08-06 04:51:23 on dist-test-slave-1x4g"
server_key: "7ff3302e4461c9a953fcd5429dcdc358"
server_key_iv: "6fdd479b8c9377fa7c2d5fcbb764632f"
server_key_version: "encryptionkey@0"
I20260806 04:51:23.525898 15408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:23.543320 15408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:23.546655 15408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:23.546880 15408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:23.549495 15408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:23.549600 15408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:23.549747 15408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:23.549825 15408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:51:23 dist-test-slave-1x4g krb5kdc[15007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991883, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:23.572582 15102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.549797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:44065 (local address 127.13.148.190:44125)
0806 04:51:23.549975 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:23.549981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:23.551273 (+ 1292us) server_negotiation.cc:407] Connection header received
0806 04:51:23.553228 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:23.553233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:23.553316 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:23.553467 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:23.556132 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.557148 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.560195 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.560633 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.560792 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.565618 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:23.565622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:23.565625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:23.565654 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:23.568930 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:23.569763 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:23.569769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:23.569772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:23.569922 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:23.570423 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:23.570429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:23.570435 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:23.570880 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:23.571070 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:23.571833 (+ 763us) server_negotiation.cc:299] Negotiation successful
0806 04:51:23.572082 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:51:23.573302 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.550296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:44125 (local address 127.13.148.131:44065)
0806 04:51:23.551104 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:23.551149 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:23.552971 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:23.553918 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:23.553934 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:23.554983 (+ 1049us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:23.555907 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.555942 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.557383 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.557390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.560033 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.560052 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.560885 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.560898 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.561203 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.561980 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:23.562005 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:23.565409 (+ 3404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:23.569056 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:23.569065 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:23.569079 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:23.569582 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:23.570040 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:23.570044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:23.570048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:23.570300 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:23.571047 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:23.571056 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:23.571354 (+ 298us) client_negotiation.cc:769] Sending connection context
0806 04:51:23.571716 (+ 362us) client_negotiation.cc:240] Negotiation successful
0806 04:51:23.572172 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":349,"threads_started":1}
I20260806 04:51:23.591738 15102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.578580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:46657 (local address 127.13.148.190:44125)
0806 04:51:23.578771 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:23.578778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:23.579411 (+ 633us) server_negotiation.cc:407] Connection header received
0806 04:51:23.579576 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:23.579580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:23.579628 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:23.579693 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:23.581044 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.581854 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.583499 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.584083 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.584214 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.585943 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:23.585949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:23.585962 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:23.586017 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:23.589344 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:23.589952 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:23.589957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:23.589960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:23.590106 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:23.590461 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:23.590465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:23.590468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:23.590709 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:23.590861 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:23.591229 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 04:51:23.591400 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:51:23.591795 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.578561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:44125 (local address 127.13.148.131:46657)
0806 04:51:23.579306 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:23.579326 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:23.579475 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:23.579752 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:23.579758 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:23.580327 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:23.580821 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.580841 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.581982 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.581987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.583371 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.583381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.584274 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.584281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.584409 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.585044 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:23.585087 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:23.585755 (+ 668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:23.589472 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:23.589479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:23.589485 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:23.589837 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:23.590210 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:23.590215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:23.590218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:23.590335 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:23.590819 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:23.590825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:23.590991 (+ 166us) client_negotiation.cc:769] Sending connection context
0806 04:51:23.591142 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 04:51:23.591408 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":383,"threads_started":1}
I20260806 04:51:23.598150 15408 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:44239
I20260806 04:51:23.598188 15546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:44239 every 8 connection(s)
I20260806 04:51:23.599236 15408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:51:23.599556 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 15408
I20260806 04:51:23.599723 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:51:23.600271 13906 external_mini_cluster.cc:1428] Setting key 55d91a046e4be38379d6ff68b7e7e972
I20260806 04:51:23.601114 15547 heartbeater.cc:347] Connected to a master server at 127.13.148.190:44125
I20260806 04:51:23.601327 15547 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:23.601963 15547 heartbeater.cc:511] Master 127.13.148.190:44125 requested a full tablet report, sending...
I20260806 04:51:23.603137 15052 ts_manager.cc:194] Registered new tserver with Master: b726e3e7aa6b434da437b7840c219baf (127.13.148.131:44239)
I20260806 04:51:23.603796 15052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:44065
I20260806 04:51:23.614712 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 04:51:23 dist-test-slave-1x4g krb5kdc[15007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991883, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 04:51:23 dist-test-slave-1x4g krb5kdc[15007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991883, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:23.657689 15102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.644987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.13.148.190:44125)
0806 04:51:23.645132 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:23.645136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:23.645468 (+ 332us) server_negotiation.cc:407] Connection header received
0806 04:51:23.645595 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:23.645598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:23.645665 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:23.645739 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:23.646876 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.647578 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.648807 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.649466 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.649583 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.653184 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:23.653189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:23.653192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:23.653224 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:23.655496 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:23.656016 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:23.656019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:23.656021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:23.656131 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:23.656413 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:23.656416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:23.656417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:23.656690 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:23.656914 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:23.657163 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 04:51:23.657342 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:51:23.657677 15263 heartbeater.cc:503] Master 127.13.148.190:44125 was elected leader, sending a full tablet report...
I20260806 04:51:23.661319 15053 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:51:23.667384 15053 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:51:23.688549 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.678204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50324 (local address 127.13.148.131:44239)
0806 04:51:23.679438 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:23.679447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:23.679932 (+ 485us) server_negotiation.cc:407] Connection header received
0806 04:51:23.680361 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:23.680372 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:23.680619 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:23.680786 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:23.681778 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.683256 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.685749 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.687035 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.687199 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.687385 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:23.688048 (+ 663us) server_negotiation.cc:299] Negotiation successful
0806 04:51:23.688209 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1057,"thread_start_us":871,"threads_started":1}
I20260806 04:51:23.689396 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.677981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:44239 (local address 127.0.0.1:50324)
0806 04:51:23.679787 (+ 1806us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:23.679838 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:23.680088 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:23.681160 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:23.681167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:23.681194 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:23.681574 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.681587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.683430 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.683435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.685588 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.685602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.687579 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.687585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.687784 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.687834 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 04:51:23.688903 (+ 1069us) client_negotiation.cc:240] Negotiation successful
0806 04:51:23.689092 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1573,"thread_start_us":1419,"threads_started":1}
I20260806 04:51:23.690737 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.678620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:43873 (local address 127.0.0.1:46840)
0806 04:51:23.680345 (+ 1725us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:23.680366 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:23.680458 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:23.682274 (+ 1816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:23.682279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:23.682299 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:23.682648 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.682658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.685259 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.685264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.687206 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.687220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.690015 (+ 2795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.690020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.690193 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.690224 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 04:51:23.690319 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 04:51:23.690438 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1595,"thread_start_us":1447,"threads_started":1}
I20260806 04:51:23.690743 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.679237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.13.148.129:43873)
0806 04:51:23.679940 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:23.679947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:23.680573 (+ 626us) server_negotiation.cc:407] Connection header received
0806 04:51:23.680664 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:23.680670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:23.681902 (+ 1232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:23.682088 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:23.682815 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.684184 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.687385 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.689799 (+ 2414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.689962 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.690123 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:23.690371 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 04:51:23.690526 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":422,"threads_started":1}
I20260806 04:51:23.693965 15475 tablet_service.cc:1467] Processing CreateTablet for tablet 0d21e0769ba7435f9b2c0ea06d493482 (DEFAULT_TABLE table=test-table [id=83513663f785473c8b12c93d17266ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:23.694092 15190 tablet_service.cc:1467] Processing CreateTablet for tablet 0d21e0769ba7435f9b2c0ea06d493482 (DEFAULT_TABLE table=test-table [id=83513663f785473c8b12c93d17266ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:23.695251 15475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d21e0769ba7435f9b2c0ea06d493482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:23.695309 15190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d21e0769ba7435f9b2c0ea06d493482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:23.696035 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.680229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:40121 (local address 127.0.0.1:39366)
0806 04:51:23.689669 (+ 9440us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:23.689684 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:23.689767 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:23.690415 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:23.690419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:23.690450 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:23.690596 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.690606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.692067 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.692075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.693776 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.693789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.695429 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.695434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.695600 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.695626 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:51:23.695714 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 04:51:23.695828 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9364,"thread_start_us":9894,"threads_started":1}
I20260806 04:51:23.696151 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.680633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.13.148.130:40121)
0806 04:51:23.689937 (+ 9304us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:23.689944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:23.689967 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:51:23.690053 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:23.690058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:23.690244 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:23.690757 (+ 513us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:23.690940 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.691909 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.694103 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.695224 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.695390 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.695540 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:23.695774 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 04:51:23.695907 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9158,"thread_start_us":9042,"threads_started":1}
I20260806 04:51:23.702328 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 0d21e0769ba7435f9b2c0ea06d493482 (DEFAULT_TABLE table=test-table [id=83513663f785473c8b12c93d17266ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:23.703516 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d21e0769ba7435f9b2c0ea06d493482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:23.705377 15566 tablet_bootstrap.cc:492] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8: Bootstrap starting.
I20260806 04:51:23.707274 15567 tablet_bootstrap.cc:492] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf: Bootstrap starting.
I20260806 04:51:23.709939 15566 tablet_bootstrap.cc:654] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:23.710628 15567 tablet_bootstrap.cc:654] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:23.711236 15566 log.cc:824] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:23.711915 15567 log.cc:824] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:23.714217 15566 tablet_bootstrap.cc:492] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8: No bootstrap required, opened a new log
I20260806 04:51:23.714224 15567 tablet_bootstrap.cc:492] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf: No bootstrap required, opened a new log
I20260806 04:51:23.714457 15566 ts_tablet_manager.cc:1397] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260806 04:51:23.714469 15567 ts_tablet_manager.cc:1397] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260806 04:51:23.715889 15570 tablet_bootstrap.cc:492] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1: Bootstrap starting.
I20260806 04:51:23.719286 15570 tablet_bootstrap.cc:654] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:23.720347 15570 log.cc:824] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:23.721912 15566 raft_consensus.cc:348] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } }
I20260806 04:51:23.722231 15566 raft_consensus.cc:374] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:23.722353 15566 raft_consensus.cc:729] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190658c7116946e3ba221898fdd4d4c8, State: Initialized, Role: FOLLOWER
I20260806 04:51:23.722853 15570 tablet_bootstrap.cc:492] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1: No bootstrap required, opened a new log
I20260806 04:51:23.722898 15566 consensus_queue.cc:260] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } }
I20260806 04:51:23.723080 15570 ts_tablet_manager.cc:1397] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260806 04:51:23.723865 15567 raft_consensus.cc:348] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } }
I20260806 04:51:23.724262 15567 raft_consensus.cc:374] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:23.724421 15567 raft_consensus.cc:729] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b726e3e7aa6b434da437b7840c219baf, State: Initialized, Role: FOLLOWER
I20260806 04:51:23.725005 15567 consensus_queue.cc:260] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } }
I20260806 04:51:23.726258 15547 heartbeater.cc:503] Master 127.13.148.190:44125 was elected leader, sending a full tablet report...
I20260806 04:51:23.726769 15567 ts_tablet_manager.cc:1428] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260806 04:51:23.728256 15566 ts_tablet_manager.cc:1428] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260806 04:51:23.733187 15570 raft_consensus.cc:348] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } }
I20260806 04:51:23.733467 15570 raft_consensus.cc:374] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:23.733566 15570 raft_consensus.cc:729] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709f94a1781b4c7896cbf3e366bc19e1, State: Initialized, Role: FOLLOWER
I20260806 04:51:23.733980 15570 consensus_queue.cc:260] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } }
I20260806 04:51:23.734773 15404 heartbeater.cc:503] Master 127.13.148.190:44125 was elected leader, sending a full tablet report...
I20260806 04:51:23.735092 15570 ts_tablet_manager.cc:1428] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260806 04:51:23.751089 15572 raft_consensus.cc:482] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:51:23.751312 15572 raft_consensus.cc:504] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } }
I20260806 04:51:23.752583 15572 leader_election.cc:288] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b726e3e7aa6b434da437b7840c219baf (127.13.148.131:44239), 709f94a1781b4c7896cbf3e366bc19e1 (127.13.148.130:40121)
I20260806 04:51:23.759321 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.753827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:38681 (local address 127.13.148.130:40121)
0806 04:51:23.754008 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:23.754014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:23.754278 (+ 264us) server_negotiation.cc:407] Connection header received
0806 04:51:23.754447 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:23.754459 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:23.754517 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:23.754600 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:23.755257 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.755928 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.757684 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.758461 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.758566 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.758663 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:23.758983 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 04:51:23.759096 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:51:23.759385 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.752970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:46109 (local address 127.13.148.131:44239)
0806 04:51:23.753177 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:23.753186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:23.753671 (+ 485us) server_negotiation.cc:407] Connection header received
0806 04:51:23.753990 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:23.753995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:23.754066 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:23.754128 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:23.755007 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.755789 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.757760 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.758501 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.758662 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.758723 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:23.759048 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 04:51:23.759158 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:51:23.759496 15575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.752903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:44239 (local address 127.13.148.129:46109)
0806 04:51:23.753581 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:23.753600 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:23.753782 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:23.754319 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:23.754326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:23.754353 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:23.754855 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.754866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.755911 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.755915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.757570 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.757582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.758725 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.758729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.758876 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.758896 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 04:51:23.758994 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 04:51:23.759099 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":84,"thread_start_us":301,"threads_started":1}
I20260806 04:51:23.759665 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.753643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:40121 (local address 127.13.148.129:38681)
0806 04:51:23.754185 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:23.754198 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:23.754333 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:23.754617 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:23.754621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:23.754643 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:23.755121 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.755133 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.756072 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.756076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.757534 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:23.757546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.758648 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.758654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:23.758842 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.758864 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:51:23.758929 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 04:51:23.759066 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":21,"thread_start_us":310,"threads_started":1}
I20260806 04:51:23.760346 15496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d21e0769ba7435f9b2c0ea06d493482" candidate_uuid: "190658c7116946e3ba221898fdd4d4c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b726e3e7aa6b434da437b7840c219baf" is_pre_election: true
I20260806 04:51:23.760480 15358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d21e0769ba7435f9b2c0ea06d493482" candidate_uuid: "190658c7116946e3ba221898fdd4d4c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709f94a1781b4c7896cbf3e366bc19e1" is_pre_election: true
I20260806 04:51:23.760793 15496 raft_consensus.cc:2454] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 190658c7116946e3ba221898fdd4d4c8 in term 0.
I20260806 04:51:23.760965 15358 raft_consensus.cc:2454] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 190658c7116946e3ba221898fdd4d4c8 in term 0.
I20260806 04:51:23.761312 15144 leader_election.cc:302] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 190658c7116946e3ba221898fdd4d4c8, b726e3e7aa6b434da437b7840c219baf; no voters:
I20260806 04:51:23.761596 15572 raft_consensus.cc:2781] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:51:23.761718 15572 raft_consensus.cc:482] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:51:23.761817 15572 raft_consensus.cc:3037] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:23.763692 15572 raft_consensus.cc:504] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } }
I20260806 04:51:23.764206 15572 leader_election.cc:288] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [CANDIDATE]: Term 1 election: Requested vote from peers b726e3e7aa6b434da437b7840c219baf (127.13.148.131:44239), 709f94a1781b4c7896cbf3e366bc19e1 (127.13.148.130:40121)
I20260806 04:51:23.764535 15496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d21e0769ba7435f9b2c0ea06d493482" candidate_uuid: "190658c7116946e3ba221898fdd4d4c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b726e3e7aa6b434da437b7840c219baf"
I20260806 04:51:23.764621 15358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d21e0769ba7435f9b2c0ea06d493482" candidate_uuid: "190658c7116946e3ba221898fdd4d4c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709f94a1781b4c7896cbf3e366bc19e1"
I20260806 04:51:23.764703 15496 raft_consensus.cc:3037] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:23.764820 15358 raft_consensus.cc:3037] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:23.766618 15496 raft_consensus.cc:2454] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 190658c7116946e3ba221898fdd4d4c8 in term 1.
I20260806 04:51:23.766624 15358 raft_consensus.cc:2454] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 190658c7116946e3ba221898fdd4d4c8 in term 1.
I20260806 04:51:23.766927 15144 leader_election.cc:302] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 190658c7116946e3ba221898fdd4d4c8, b726e3e7aa6b434da437b7840c219baf; no voters:
I20260806 04:51:23.767189 15572 raft_consensus.cc:2781] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:23.767618 15572 raft_consensus.cc:686] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [term 1 LEADER]: Becoming Leader. State: Replica: 190658c7116946e3ba221898fdd4d4c8, State: Running, Role: LEADER
I20260806 04:51:23.768031 15572 consensus_queue.cc:237] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } }
I20260806 04:51:23.772078 15053 catalog_manager.cc:5495] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 190658c7116946e3ba221898fdd4d4c8 (127.13.148.129). New cstate: current_term: 1 leader_uuid: "190658c7116946e3ba221898fdd4d4c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } health_report { overall_health: HEALTHY } } }
I20260806 04:51:23.817440 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:23.812858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46852 (local address 127.13.148.129:43873)
0806 04:51:23.813023 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:23.813027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:23.813150 (+ 123us) server_negotiation.cc:407] Connection header received
0806 04:51:23.813385 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:23.813388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:23.813461 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:23.813564 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:51:23.813837 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.814418 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.815544 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:23.815882 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:23.815960 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:23.816336 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:51:23.816895 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:51:23.817005 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:23.817111 (+ 106us) server_negotiation.cc:299] Negotiation successful
0806 04:51:23.817173 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:51:23.824532 15496 raft_consensus.cc:1261] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf [term 1 FOLLOWER]: Refusing update from remote peer 190658c7116946e3ba221898fdd4d4c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:23.824530 15358 raft_consensus.cc:1261] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1 [term 1 FOLLOWER]: Refusing update from remote peer 190658c7116946e3ba221898fdd4d4c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:23.825426 15578 consensus_queue.cc:1035] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:23.825757 15572 consensus_queue.cc:1035] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:23.836165 15581 mvcc.cc:205] Tried to move back new op lower bound from 7315422756137095168 to 7315422755917053952. Current Snapshot: MvccSnapshot[applied={T|T < 7315422756137095168}]
I20260806 04:51:23.836426 15582 mvcc.cc:205] Tried to move back new op lower bound from 7315422756137095168 to 7315422755917053952. Current Snapshot: MvccSnapshot[applied={T|T < 7315422756137095168}]
I20260806 04:51:23.853933 15053 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:51:23.854202 15053 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:51:23.857502 15053 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df: Sending DeleteTablet for 3 replicas of tablet 0d21e0769ba7435f9b2c0ea06d493482
I20260806 04:51:23.858780 15475 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d21e0769ba7435f9b2c0ea06d493482 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:23 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:50324
I20260806 04:51:23.858805 15332 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d21e0769ba7435f9b2c0ea06d493482 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:23 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:39366
I20260806 04:51:23.858862 15190 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d21e0769ba7435f9b2c0ea06d493482 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:23 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:46840
I20260806 04:51:23.859654 15590 tablet_replica.cc:330] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1: stopping tablet replica
I20260806 04:51:23.860024 15590 raft_consensus.cc:2229] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:51:23.860427 15590 raft_consensus.cc:2258] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:23.861107 15053 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:51:23.861851 15053 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:51:23.864168 15590 ts_tablet_manager.cc:1905] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:23.870926 15592 tablet_replica.cc:330] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf: stopping tablet replica
I20260806 04:51:23.871273 15592 raft_consensus.cc:2229] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:51:23.871675 15592 raft_consensus.cc:2258] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:23.872035 15189 tablet_service.cc:1467] Processing CreateTablet for tablet d4f140048bf84e248c7a40bd03f7f5d3 (DEFAULT_TABLE table=test-table [id=652cff1a1ef74ce6ac6a850d54f4f6f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:23.872634 15189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4f140048bf84e248c7a40bd03f7f5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:23.872659 15332 tablet_service.cc:1467] Processing CreateTablet for tablet d4f140048bf84e248c7a40bd03f7f5d3 (DEFAULT_TABLE table=test-table [id=652cff1a1ef74ce6ac6a850d54f4f6f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:23.873075 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4f140048bf84e248c7a40bd03f7f5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:23.873857 15475 tablet_service.cc:1467] Processing CreateTablet for tablet d4f140048bf84e248c7a40bd03f7f5d3 (DEFAULT_TABLE table=test-table [id=652cff1a1ef74ce6ac6a850d54f4f6f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:23.874229 15475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4f140048bf84e248c7a40bd03f7f5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:23.874847 15592 ts_tablet_manager.cc:1905] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:23.877831 15590 ts_tablet_manager.cc:1918] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:51:23.877977 15590 log.cc:1196] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/0d21e0769ba7435f9b2c0ea06d493482
I20260806 04:51:23.878644 15590 ts_tablet_manager.cc:1939] T 0d21e0769ba7435f9b2c0ea06d493482 P 709f94a1781b4c7896cbf3e366bc19e1: Deleting consensus metadata
I20260806 04:51:23.879195 15570 tablet_bootstrap.cc:492] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1: Bootstrap starting.
I20260806 04:51:23.880268 15567 tablet_bootstrap.cc:492] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf: Bootstrap starting.
I20260806 04:51:23.880813 15570 tablet_bootstrap.cc:654] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:23.881446 15591 tablet_replica.cc:330] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8: stopping tablet replica
I20260806 04:51:23.880470 15040 catalog_manager.cc:4841] TS 709f94a1781b4c7896cbf3e366bc19e1 (127.13.148.130:40121): tablet 0d21e0769ba7435f9b2c0ea06d493482 (table test-table [id=83513663f785473c8b12c93d17266ff0]) successfully deleted
I20260806 04:51:23.881690 15567 tablet_bootstrap.cc:654] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:23.881858 15591 raft_consensus.cc:2229] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:51:23.882486 15591 raft_consensus.cc:2258] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:23.884734 15591 ts_tablet_manager.cc:1905] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:23.888492 15566 tablet_bootstrap.cc:492] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8: Bootstrap starting.
I20260806 04:51:23.889933 15566 tablet_bootstrap.cc:654] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:23.893030 15591 ts_tablet_manager.cc:1918] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:51:23.893265 15591 log.cc:1196] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/0d21e0769ba7435f9b2c0ea06d493482
I20260806 04:51:23.893889 15591 ts_tablet_manager.cc:1939] T 0d21e0769ba7435f9b2c0ea06d493482 P 190658c7116946e3ba221898fdd4d4c8: Deleting consensus metadata
I20260806 04:51:23.896373 15566 tablet_bootstrap.cc:492] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8: No bootstrap required, opened a new log
I20260806 04:51:23.896610 15566 ts_tablet_manager.cc:1397] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 04:51:23.897538 15566 raft_consensus.cc:348] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:23.897835 15566 raft_consensus.cc:374] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:23.897938 15570 tablet_bootstrap.cc:492] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1: No bootstrap required, opened a new log
I20260806 04:51:23.897992 15566 raft_consensus.cc:729] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190658c7116946e3ba221898fdd4d4c8, State: Initialized, Role: FOLLOWER
I20260806 04:51:23.898185 15570 ts_tablet_manager.cc:1397] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.005s
I20260806 04:51:23.898288 15566 consensus_queue.cc:260] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:23.899009 15570 raft_consensus.cc:348] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:23.899276 15570 raft_consensus.cc:374] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:23.899428 15570 raft_consensus.cc:729] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709f94a1781b4c7896cbf3e366bc19e1, State: Initialized, Role: FOLLOWER
I20260806 04:51:23.899708 15570 consensus_queue.cc:260] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:23.900983 15566 ts_tablet_manager.cc:1428] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20260806 04:51:23.901839 15264 tablet.cc:2369] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:51:23.902623 15037 catalog_manager.cc:4841] TS 190658c7116946e3ba221898fdd4d4c8 (127.13.148.129:43873): tablet 0d21e0769ba7435f9b2c0ea06d493482 (table test-table [id=83513663f785473c8b12c93d17266ff0]) successfully deleted
I20260806 04:51:23.903743 15570 ts_tablet_manager.cc:1428] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260806 04:51:23.904632 15592 ts_tablet_manager.cc:1918] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:51:23.904912 15592 log.cc:1196] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/0d21e0769ba7435f9b2c0ea06d493482
I20260806 04:51:23.905594 15592 ts_tablet_manager.cc:1939] T 0d21e0769ba7435f9b2c0ea06d493482 P b726e3e7aa6b434da437b7840c219baf: Deleting consensus metadata
I20260806 04:51:23.906529 15567 tablet_bootstrap.cc:492] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf: No bootstrap required, opened a new log
I20260806 04:51:23.906713 15567 ts_tablet_manager.cc:1397] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.011s
I20260806 04:51:23.907475 15040 catalog_manager.cc:4841] TS b726e3e7aa6b434da437b7840c219baf (127.13.148.131:44239): tablet 0d21e0769ba7435f9b2c0ea06d493482 (table test-table [id=83513663f785473c8b12c93d17266ff0]) successfully deleted
I20260806 04:51:23.907536 15567 raft_consensus.cc:348] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:23.907789 15567 raft_consensus.cc:374] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:23.907918 15567 raft_consensus.cc:729] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b726e3e7aa6b434da437b7840c219baf, State: Initialized, Role: FOLLOWER
I20260806 04:51:23.908206 15567 consensus_queue.cc:260] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:23.908826 15567 ts_tablet_manager.cc:1428] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20260806 04:51:24.101620 15548 tablet.cc:2369] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:51:24.121976 15572 raft_consensus.cc:482] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:51:24.122156 15572 raft_consensus.cc:504] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:24.122777 15572 leader_election.cc:288] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 709f94a1781b4c7896cbf3e366bc19e1 (127.13.148.130:40121), b726e3e7aa6b434da437b7840c219baf (127.13.148.131:44239)
I20260806 04:51:24.123504 15496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f140048bf84e248c7a40bd03f7f5d3" candidate_uuid: "190658c7116946e3ba221898fdd4d4c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b726e3e7aa6b434da437b7840c219baf" is_pre_election: true
I20260806 04:51:24.123502 15358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f140048bf84e248c7a40bd03f7f5d3" candidate_uuid: "190658c7116946e3ba221898fdd4d4c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709f94a1781b4c7896cbf3e366bc19e1" is_pre_election: true
I20260806 04:51:24.123754 15496 raft_consensus.cc:2454] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 190658c7116946e3ba221898fdd4d4c8 in term 0.
I20260806 04:51:24.123785 15358 raft_consensus.cc:2454] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 190658c7116946e3ba221898fdd4d4c8 in term 0.
I20260806 04:51:24.124270 15144 leader_election.cc:302] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 190658c7116946e3ba221898fdd4d4c8, b726e3e7aa6b434da437b7840c219baf; no voters:
I20260806 04:51:24.124541 15572 raft_consensus.cc:2781] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:51:24.124696 15572 raft_consensus.cc:482] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:51:24.124843 15572 raft_consensus.cc:3037] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:24.126817 15572 raft_consensus.cc:504] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:24.127389 15572 leader_election.cc:288] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [CANDIDATE]: Term 1 election: Requested vote from peers 709f94a1781b4c7896cbf3e366bc19e1 (127.13.148.130:40121), b726e3e7aa6b434da437b7840c219baf (127.13.148.131:44239)
I20260806 04:51:24.127813 15358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f140048bf84e248c7a40bd03f7f5d3" candidate_uuid: "190658c7116946e3ba221898fdd4d4c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709f94a1781b4c7896cbf3e366bc19e1"
I20260806 04:51:24.127915 15496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f140048bf84e248c7a40bd03f7f5d3" candidate_uuid: "190658c7116946e3ba221898fdd4d4c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b726e3e7aa6b434da437b7840c219baf"
I20260806 04:51:24.128010 15358 raft_consensus.cc:3037] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:24.128111 15496 raft_consensus.cc:3037] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:24.130234 15358 raft_consensus.cc:2454] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 190658c7116946e3ba221898fdd4d4c8 in term 1.
I20260806 04:51:24.130872 15496 raft_consensus.cc:2454] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 190658c7116946e3ba221898fdd4d4c8 in term 1.
I20260806 04:51:24.131225 15144 leader_election.cc:302] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 190658c7116946e3ba221898fdd4d4c8, 709f94a1781b4c7896cbf3e366bc19e1; no voters:
I20260806 04:51:24.131505 15572 raft_consensus.cc:2781] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:24.131664 15572 raft_consensus.cc:686] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [term 1 LEADER]: Becoming Leader. State: Replica: 190658c7116946e3ba221898fdd4d4c8, State: Running, Role: LEADER
I20260806 04:51:24.131945 15572 consensus_queue.cc:237] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:24.134863 15053 catalog_manager.cc:5495] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 190658c7116946e3ba221898fdd4d4c8 (127.13.148.129). New cstate: current_term: 1 leader_uuid: "190658c7116946e3ba221898fdd4d4c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } health_report { overall_health: UNKNOWN } } }
W20260806 04:51:24.138276 15405 tablet.cc:2369] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 04:51:24 dist-test-slave-1x4g krb5kdc[15007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991882, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:24.211931 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:24.198753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:44125 (local address 127.0.0.1:52310)
0806 04:51:24.199202 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:24.199247 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:24.199404 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:24.199755 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:24.199760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:24.200457 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:24.200674 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:24.200683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.201716 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.201721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:24.202649 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:24.202660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.203356 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.203360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:24.203530 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:24.204120 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:24.204135 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:24.206431 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:24.209361 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:24.209374 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:24.209380 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:24.209725 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:24.210120 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:24.210123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:24.210143 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:24.210291 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:24.210799 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:24.210807 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:24.211023 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 04:51:24.211183 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 04:51:24.211451 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":233,"threads_started":1}
I20260806 04:51:24.211951 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:24.198804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52310 (local address 127.13.148.190:44125)
0806 04:51:24.199519 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:24.199524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:24.199540 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:51:24.199583 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:24.199586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:24.199632 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:24.199732 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:24.200860 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.201596 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.202788 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.203207 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.203315 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:24.206620 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:24.206624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:24.206627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:24.206651 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:24.209251 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:24.209881 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:24.209886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:24.209891 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:24.210000 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:24.210390 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:24.210396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:24.210399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:24.210684 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:24.210839 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:24.211263 (+ 424us) server_negotiation.cc:299] Negotiation successful
0806 04:51:24.211577 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":571,"thread_start_us":474,"threads_started":1}
I20260806 04:51:24.217269 15053 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:52310:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 04:51:24.218462 15053 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:51:24.224740 15189 tablet_service.cc:1467] Processing CreateTablet for tablet c6cf7ab6c3b148f1a4cd08c1915a156e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a7dac92288ce4bd9bb1b0ace3a65daa6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:51:24.224779 15332 tablet_service.cc:1467] Processing CreateTablet for tablet c6cf7ab6c3b148f1a4cd08c1915a156e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a7dac92288ce4bd9bb1b0ace3a65daa6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:51:24.225040 15475 tablet_service.cc:1467] Processing CreateTablet for tablet c6cf7ab6c3b148f1a4cd08c1915a156e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a7dac92288ce4bd9bb1b0ace3a65daa6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:51:24.225266 15189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6cf7ab6c3b148f1a4cd08c1915a156e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:24.225338 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6cf7ab6c3b148f1a4cd08c1915a156e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:24.225500 15475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6cf7ab6c3b148f1a4cd08c1915a156e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:24.232858 15566 tablet_bootstrap.cc:492] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8: Bootstrap starting.
I20260806 04:51:24.232908 15567 tablet_bootstrap.cc:492] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf: Bootstrap starting.
I20260806 04:51:24.234431 15567 tablet_bootstrap.cc:654] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:24.234565 15566 tablet_bootstrap.cc:654] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:24.236239 15570 tablet_bootstrap.cc:492] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1: Bootstrap starting.
I20260806 04:51:24.237838 15570 tablet_bootstrap.cc:654] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:24.239450 15567 tablet_bootstrap.cc:492] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf: No bootstrap required, opened a new log
I20260806 04:51:24.239645 15567 ts_tablet_manager.cc:1397] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20260806 04:51:24.240279 15567 raft_consensus.cc:348] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:24.240494 15567 raft_consensus.cc:374] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:24.240582 15567 raft_consensus.cc:729] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b726e3e7aa6b434da437b7840c219baf, State: Initialized, Role: FOLLOWER
I20260806 04:51:24.240886 15567 consensus_queue.cc:260] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:24.241415 15573 tablet_replica.cc:439] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } }
I20260806 04:51:24.241690 15573 tablet_replica.cc:442] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:51:24.242728 15567 ts_tablet_manager.cc:1428] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:51:24.244475 15570 tablet_bootstrap.cc:492] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1: No bootstrap required, opened a new log
I20260806 04:51:24.244616 15570 ts_tablet_manager.cc:1397] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260806 04:51:24.245231 15570 raft_consensus.cc:348] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:24.245448 15570 raft_consensus.cc:374] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:24.245503 15570 raft_consensus.cc:729] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709f94a1781b4c7896cbf3e366bc19e1, State: Initialized, Role: FOLLOWER
I20260806 04:51:24.245692 15570 consensus_queue.cc:260] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:24.245903 15566 tablet_bootstrap.cc:492] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8: No bootstrap required, opened a new log
I20260806 04:51:24.246078 15566 ts_tablet_manager.cc:1397] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20260806 04:51:24.246138 15570 ts_tablet_manager.cc:1428] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260806 04:51:24.246461 15574 tablet_replica.cc:439] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } }
I20260806 04:51:24.246752 15574 tablet_replica.cc:442] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:51:24.246837 15566 raft_consensus.cc:348] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:24.247105 15566 raft_consensus.cc:374] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:24.247196 15566 raft_consensus.cc:729] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190658c7116946e3ba221898fdd4d4c8, State: Initialized, Role: FOLLOWER
I20260806 04:51:24.247486 15566 consensus_queue.cc:260] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:24.247939 15572 tablet_replica.cc:439] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } }
I20260806 04:51:24.248203 15566 ts_tablet_manager.cc:1428] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260806 04:51:24.248283 15572 tablet_replica.cc:442] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:51:24.506589 15574 raft_consensus.cc:482] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:51:24.506850 15574 raft_consensus.cc:504] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:24.508447 15574 leader_election.cc:288] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 190658c7116946e3ba221898fdd4d4c8 (127.13.148.129:43873), b726e3e7aa6b434da437b7840c219baf (127.13.148.131:44239)
I20260806 04:51:24.526603 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:24.508703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:57861 (local address 127.13.148.129:43873)
0806 04:51:24.521211 (+ 12508us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:24.521221 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:24.521763 (+ 542us) server_negotiation.cc:407] Connection header received
0806 04:51:24.521863 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:24.521868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:24.521951 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:24.522077 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:24.522526 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.523166 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.524989 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.525703 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.525862 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:24.525934 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:24.526198 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 04:51:24.526320 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12326,"thread_start_us":12178,"threads_started":1}
I20260806 04:51:24.526665 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:24.508646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:43873 (local address 127.13.148.130:57861)
0806 04:51:24.521411 (+ 12765us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:24.521437 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:24.521618 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:24.522093 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:24.522098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:24.522122 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:24.522366 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:24.522375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.523292 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.523298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:24.524839 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:24.524859 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.525852 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.525857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:24.526039 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:24.526060 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 04:51:24.526165 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 04:51:24.526289 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12653,"mutex_wait_us":59,"thread_start_us":12550,"threads_started":1}
I20260806 04:51:24.526758 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:24.509028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:54423 (local address 127.13.148.131:44239)
0806 04:51:24.522052 (+ 13024us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:24.522057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:24.522072 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:51:24.522131 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:24.522135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:24.522190 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:24.522315 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:24.522846 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.523504 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.525184 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.525885 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.526062 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:24.526123 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:24.526382 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 04:51:24.526515 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12925,"thread_start_us":12816,"threads_started":1}
I20260806 04:51:24.527511 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:24.508710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:44239 (local address 127.13.148.130:54423)
0806 04:51:24.521758 (+ 13048us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:24.521782 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:24.521881 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:24.522336 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:24.522341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:24.522373 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:24.522694 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:24.522702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.523633 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.523639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:24.525032 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:24.525044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.526074 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.526079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:24.526233 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:24.526255 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:51:24.526345 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 04:51:24.526470 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12904,"thread_start_us":12789,"threads_started":1}
I20260806 04:51:24.527418 15211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6cf7ab6c3b148f1a4cd08c1915a156e" candidate_uuid: "709f94a1781b4c7896cbf3e366bc19e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "190658c7116946e3ba221898fdd4d4c8" is_pre_election: true
I20260806 04:51:24.527815 15211 raft_consensus.cc:2454] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 709f94a1781b4c7896cbf3e366bc19e1 in term 0.
I20260806 04:51:24.528167 15496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6cf7ab6c3b148f1a4cd08c1915a156e" candidate_uuid: "709f94a1781b4c7896cbf3e366bc19e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b726e3e7aa6b434da437b7840c219baf" is_pre_election: true
I20260806 04:51:24.528415 15496 raft_consensus.cc:2454] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 709f94a1781b4c7896cbf3e366bc19e1 in term 0.
I20260806 04:51:24.528405 15283 leader_election.cc:302] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 190658c7116946e3ba221898fdd4d4c8, 709f94a1781b4c7896cbf3e366bc19e1; no voters:
I20260806 04:51:24.528827 15574 raft_consensus.cc:2781] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:51:24.528945 15574 raft_consensus.cc:482] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:51:24.529057 15574 raft_consensus.cc:3037] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:24.530990 15574 raft_consensus.cc:504] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:24.531606 15574 leader_election.cc:288] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [CANDIDATE]: Term 1 election: Requested vote from peers 190658c7116946e3ba221898fdd4d4c8 (127.13.148.129:43873), b726e3e7aa6b434da437b7840c219baf (127.13.148.131:44239)
I20260806 04:51:24.532012 15211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6cf7ab6c3b148f1a4cd08c1915a156e" candidate_uuid: "709f94a1781b4c7896cbf3e366bc19e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "190658c7116946e3ba221898fdd4d4c8"
I20260806 04:51:24.532101 15496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6cf7ab6c3b148f1a4cd08c1915a156e" candidate_uuid: "709f94a1781b4c7896cbf3e366bc19e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b726e3e7aa6b434da437b7840c219baf"
I20260806 04:51:24.532196 15211 raft_consensus.cc:3037] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:24.532366 15496 raft_consensus.cc:3037] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:24.534443 15211 raft_consensus.cc:2454] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 709f94a1781b4c7896cbf3e366bc19e1 in term 1.
I20260806 04:51:24.534471 15496 raft_consensus.cc:2454] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 709f94a1781b4c7896cbf3e366bc19e1 in term 1.
I20260806 04:51:24.534793 15283 leader_election.cc:302] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 190658c7116946e3ba221898fdd4d4c8, 709f94a1781b4c7896cbf3e366bc19e1; no voters:
I20260806 04:51:24.535112 15574 raft_consensus.cc:2781] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:24.535491 15574 raft_consensus.cc:686] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [term 1 LEADER]: Becoming Leader. State: Replica: 709f94a1781b4c7896cbf3e366bc19e1, State: Running, Role: LEADER
I20260806 04:51:24.535811 15574 consensus_queue.cc:237] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } }
I20260806 04:51:24.537667 15611 tablet_replica.cc:439] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1: TxnStatusTablet state changed. Reason: New leader 709f94a1781b4c7896cbf3e366bc19e1. Latest consensus state: current_term: 1 leader_uuid: "709f94a1781b4c7896cbf3e366bc19e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } }
I20260806 04:51:24.538101 15611 tablet_replica.cc:442] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:51:24.539410 15053 catalog_manager.cc:5495] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 709f94a1781b4c7896cbf3e366bc19e1 (127.13.148.130). New cstate: current_term: 1 leader_uuid: "709f94a1781b4c7896cbf3e366bc19e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } health_report { overall_health: UNKNOWN } } }
I20260806 04:51:24.539731 15613 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 04:51:24.539969 15613 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 04:51:24.566406 15572 consensus_queue.cc:1035] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:24.572319 15572 consensus_queue.cc:1035] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:24.615370 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:24.607556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:40121 (local address 127.0.0.1:39378)
0806 04:51:24.607906 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:24.607925 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:24.608051 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:24.609116 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:24.609121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:24.609136 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:51:24.609342 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:24.609352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.610823 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.610828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:24.612430 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:24.612446 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.613473 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.613479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:24.613682 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:24.613753 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:51:24.614799 (+ 1046us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:51:24.614820 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 04:51:24.614900 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 04:51:24.615043 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260806 04:51:24.615404 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:24.607547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.13.148.130:40121)
0806 04:51:24.608649 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:24.608654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:24.608673 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:51:24.608757 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:24.608761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:24.608926 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:24.609063 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:51:24.609521 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.610652 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.612627 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.613319 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.613436 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:24.613986 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:51:24.614691 (+ 705us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:51:24.614807 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:24.614952 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 04:51:24.615030 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":973,"thread_start_us":783,"threads_started":1}
I20260806 04:51:24.619951 15496 raft_consensus.cc:1261] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf [term 1 FOLLOWER]: Refusing update from remote peer 709f94a1781b4c7896cbf3e366bc19e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:24.620484 15211 raft_consensus.cc:1261] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8 [term 1 FOLLOWER]: Refusing update from remote peer 709f94a1781b4c7896cbf3e366bc19e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:24.620641 15611 consensus_queue.cc:1035] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:24.621317 15611 consensus_queue.cc:1035] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:24.629545 15573 tablet_replica.cc:439] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf: TxnStatusTablet state changed. Reason: New leader 709f94a1781b4c7896cbf3e366bc19e1. Latest consensus state: current_term: 1 leader_uuid: "709f94a1781b4c7896cbf3e366bc19e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } }
I20260806 04:51:24.629807 15573 tablet_replica.cc:442] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:51:24.633205 15611 tablet_replica.cc:439] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "709f94a1781b4c7896cbf3e366bc19e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } }
I20260806 04:51:24.633457 15611 tablet_replica.cc:442] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:51:24.634063 15617 mvcc.cc:205] Tried to move back new op lower bound from 7315422759396798464 to 7315422759062044672. Current Snapshot: MvccSnapshot[applied={T|T < 7315422759396798464}]
I20260806 04:51:24.635685 15572 tablet_replica.cc:439] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8: TxnStatusTablet state changed. Reason: New leader 709f94a1781b4c7896cbf3e366bc19e1. Latest consensus state: current_term: 1 leader_uuid: "709f94a1781b4c7896cbf3e366bc19e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } }
I20260806 04:51:24.635967 15572 tablet_replica.cc:442] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:51:24.637100 15574 tablet_replica.cc:439] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "709f94a1781b4c7896cbf3e366bc19e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } }
I20260806 04:51:24.637435 15574 tablet_replica.cc:442] T c6cf7ab6c3b148f1a4cd08c1915a156e P 709f94a1781b4c7896cbf3e366bc19e1: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:51:24.638269 15573 tablet_replica.cc:439] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "709f94a1781b4c7896cbf3e366bc19e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } }
I20260806 04:51:24.638501 15573 tablet_replica.cc:442] T c6cf7ab6c3b148f1a4cd08c1915a156e P b726e3e7aa6b434da437b7840c219baf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:51:24.642411 15572 tablet_replica.cc:439] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "709f94a1781b4c7896cbf3e366bc19e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190658c7116946e3ba221898fdd4d4c8" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 43873 } } peers { permanent_uuid: "709f94a1781b4c7896cbf3e366bc19e1" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40121 } } peers { permanent_uuid: "b726e3e7aa6b434da437b7840c219baf" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44239 } } }
I20260806 04:51:24.642659 15572 tablet_replica.cc:442] T c6cf7ab6c3b148f1a4cd08c1915a156e P 190658c7116946e3ba221898fdd4d4c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:51:24.657228 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:24.650469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:41931 (local address 127.13.148.130:40121)
0806 04:51:24.650615 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:24.650620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:24.651150 (+ 530us) server_negotiation.cc:407] Connection header received
0806 04:51:24.651325 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:24.651331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:24.651399 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:24.651492 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:51:24.651989 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.653008 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.655177 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.655886 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.655997 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:24.656511 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:51:24.656640 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:51:24.656720 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:24.656880 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 04:51:24.656957 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:51:24.657545 15630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:24.650476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:40121 (local address 127.13.148.129:41931)
0806 04:51:24.651049 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:24.651065 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:24.651195 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:24.651507 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:24.651511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:24.651522 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:51:24.651848 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:24.651857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.653187 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.653193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:24.654960 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:24.654976 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.656056 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.656063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:24.656252 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:24.656329 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:51:24.656747 (+ 418us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:51:24.656758 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 04:51:24.656862 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 04:51:24.657010 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":262,"threads_started":1}
I20260806 04:51:24.698770 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:24.692756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:39919 (local address 127.13.148.129:43873)
0806 04:51:24.692956 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:24.692963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:24.693646 (+ 683us) server_negotiation.cc:407] Connection header received
0806 04:51:24.693891 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:24.693897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:24.693975 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:24.694041 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:51:24.694595 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.695238 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.697217 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.697634 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.697728 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:24.698243 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:51:24.698333 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:51:24.698409 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:24.698557 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 04:51:24.698609 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 04:51:24.699033 15633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:24.692884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:43873 (local address 127.13.148.130:39919)
0806 04:51:24.693559 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:24.693580 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:24.693744 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:24.694083 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:24.694087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:24.694100 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:51:24.694462 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:24.694473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.695350 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.695355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:24.697063 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:24.697080 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.697818 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.697825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:24.698014 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:24.698082 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:51:24.698447 (+ 365us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:51:24.698463 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 04:51:24.698535 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 04:51:24.698665 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":369,"threads_started":1}
I20260806 04:51:24.729923 15053 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:51:24.730105 15053 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:51:24.732872 15053 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0aa30ca2b61432b9b59686ee5c9f5df: Sending DeleteTablet for 3 replicas of tablet d4f140048bf84e248c7a40bd03f7f5d3
I20260806 04:51:24.733579 15189 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f140048bf84e248c7a40bd03f7f5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:24 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:46840
I20260806 04:51:24.733685 15475 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f140048bf84e248c7a40bd03f7f5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:24 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:50324
I20260806 04:51:24.734185 15636 tablet_replica.cc:330] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8: stopping tablet replica
I20260806 04:51:24.734414 15636 raft_consensus.cc:2229] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:51:24.734797 15636 raft_consensus.cc:2258] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:24.735519 15332 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f140048bf84e248c7a40bd03f7f5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:24 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:39366
I20260806 04:51:24.735754 15636 ts_tablet_manager.cc:1905] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:24.742420 15636 ts_tablet_manager.cc:1918] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:51:24.742580 15636 log.cc:1196] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/d4f140048bf84e248c7a40bd03f7f5d3
I20260806 04:51:24.742998 15636 ts_tablet_manager.cc:1939] T d4f140048bf84e248c7a40bd03f7f5d3 P 190658c7116946e3ba221898fdd4d4c8: Deleting consensus metadata
I20260806 04:51:24.743990 15037 catalog_manager.cc:4841] TS 190658c7116946e3ba221898fdd4d4c8 (127.13.148.129:43873): tablet d4f140048bf84e248c7a40bd03f7f5d3 (table test-table [id=652cff1a1ef74ce6ac6a850d54f4f6f6]) successfully deleted
I20260806 04:51:24.748238 15639 tablet_replica.cc:330] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1: stopping tablet replica
I20260806 04:51:24.748518 15639 raft_consensus.cc:2229] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:51:24.749020 15639 raft_consensus.cc:2258] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:24.749992 15639 ts_tablet_manager.cc:1905] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:24.757123 15639 ts_tablet_manager.cc:1918] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:51:24.757580 15637 tablet_replica.cc:330] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf: stopping tablet replica
I20260806 04:51:24.757778 15637 raft_consensus.cc:2229] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:51:24.758029 15637 raft_consensus.cc:2258] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:24.758280 15639 log.cc:1196] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/d4f140048bf84e248c7a40bd03f7f5d3
I20260806 04:51:24.759019 15637 ts_tablet_manager.cc:1905] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:24.761318 15639 ts_tablet_manager.cc:1939] T d4f140048bf84e248c7a40bd03f7f5d3 P 709f94a1781b4c7896cbf3e366bc19e1: Deleting consensus metadata
I20260806 04:51:24.762400 15040 catalog_manager.cc:4841] TS 709f94a1781b4c7896cbf3e366bc19e1 (127.13.148.130:40121): tablet d4f140048bf84e248c7a40bd03f7f5d3 (table test-table [id=652cff1a1ef74ce6ac6a850d54f4f6f6]) successfully deleted
Aug 06 04:51:24 dist-test-slave-1x4g krb5kdc[15007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991883, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:51:24.765151 15637 ts_tablet_manager.cc:1918] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:51:24.765280 15637 log.cc:1196] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/d4f140048bf84e248c7a40bd03f7f5d3
I20260806 04:51:24.765714 15637 ts_tablet_manager.cc:1939] T d4f140048bf84e248c7a40bd03f7f5d3 P b726e3e7aa6b434da437b7840c219baf: Deleting consensus metadata
I20260806 04:51:24.766641 15040 catalog_manager.cc:4841] TS b726e3e7aa6b434da437b7840c219baf (127.13.148.131:44239): tablet d4f140048bf84e248c7a40bd03f7f5d3 (table test-table [id=652cff1a1ef74ce6ac6a850d54f4f6f6]) successfully deleted
I20260806 04:51:24.771147 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:24.749572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46862 (local address 127.13.148.129:43873)
0806 04:51:24.749760 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:24.749767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:24.756768 (+ 7001us) server_negotiation.cc:407] Connection header received
0806 04:51:24.757307 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:24.757315 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:24.757403 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:24.757475 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:24.759258 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.759900 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.761225 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.761781 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.761888 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:24.765945 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:24.765956 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:24.765968 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:24.765994 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:24.768815 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:24.769505 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:24.769511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:24.769513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:24.769624 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:24.769973 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:24.769975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:24.769977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:24.770271 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:24.770464 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:24.770754 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 04:51:24.770888 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 04:51:24.771759 15258 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46862
I20260806 04:51:24.787843 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:24.774524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.13.148.190:44125)
0806 04:51:24.775084 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:24.775090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:24.775382 (+ 292us) server_negotiation.cc:407] Connection header received
0806 04:51:24.775544 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:24.775549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:24.775616 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:24.775708 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:24.777198 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.778425 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.779621 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.780185 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.780324 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:24.782274 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:24.782280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:24.782283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:24.782318 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:24.785389 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:24.785961 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:24.785967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:24.785970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:24.786105 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:24.786412 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:24.786417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:24.786420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:24.786766 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:24.787002 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:24.787199 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 04:51:24.787446 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":301,"threads_started":1}
W20260806 04:51:24.788450 15053 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52322
I20260806 04:51:24.789937 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 15125
I20260806 04:51:24.802796 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:24.791001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46876 (local address 127.13.148.129:43873)
0806 04:51:24.791317 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:24.791325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:24.791787 (+ 462us) server_negotiation.cc:407] Connection header received
0806 04:51:24.791883 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:24.791888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:24.791963 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:24.792055 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:24.793005 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.793949 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.794920 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.795509 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.795631 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:24.797632 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:24.797638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:24.797645 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:24.797680 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:24.800570 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:24.801139 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:24.801144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:24.801147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:24.801284 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:24.801584 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:24.801589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:24.801592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:24.801893 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:24.802121 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:24.802306 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 04:51:24.802510 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
W20260806 04:51:24.803339 15258 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46876
W20260806 04:51:24.803818 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 15125: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:51:24.803974 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 15125
I20260806 04:51:24.828992 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 15267
Aug 06 04:51:24 dist-test-slave-1x4g krb5kdc[15007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991883, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:51:24.845790 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:24.830423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39388 (local address 127.13.148.130:40121)
0806 04:51:24.830760 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:24.830769 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:24.830840 (+ 71us) server_negotiation.cc:407] Connection header received
0806 04:51:24.831143 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:24.831149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:24.831243 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:24.831353 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:24.832905 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.834128 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.835247 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.835685 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.835805 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:24.840179 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:24.840190 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:24.840203 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:24.840234 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:24.843122 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:24.843836 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:24.843841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:24.843843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:24.843963 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:24.844392 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:24.844395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:24.844397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:24.844749 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:24.845007 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:24.845328 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 04:51:24.845476 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
W20260806 04:51:24.846493 15399 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39388
W20260806 04:51:24.847164 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 15267: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:51:24.847342 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 15267
I20260806 04:51:24.867871 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 15408
Aug 06 04:51:24 dist-test-slave-1x4g krb5kdc[15007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991883, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
I20260806 04:51:24.881191 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:24.868577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.13.148.131:44239)
0806 04:51:24.868752 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:24.868757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:24.868815 (+ 58us) server_negotiation.cc:407] Connection header received
0806 04:51:24.868878 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:24.868882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:24.868943 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:24.869031 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:24.870062 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.870736 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.871813 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:24.872174 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:24.872252 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:24.876009 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:24.876022 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:24.876036 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:24.876069 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:24.879014 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:24.879555 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:24.879561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:24.879563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:24.879691 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:24.879959 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:24.879962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:24.879963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:24.880257 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:24.880456 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:24.880631 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 04:51:24.880954 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 04:51:24.881774 15541 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50336
W20260806 04:51:24.882339 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 15408: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:51:24.882474 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 15408
I20260806 04:51:24.899744 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 15020
I20260806 04:51:24.900692 15094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:25.007782 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 15020
2026-08-06T04:51:25Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4487 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:51:25 dist-test-slave-1x4g krb5kdc[15648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:51:25 dist-test-slave-1x4g krb5kdc[15648](info): set up 2 sockets
Aug 06 04:51:25 dist-test-slave-1x4g krb5kdc[15648](info): commencing operation
krb5kdc: starting...
W20260806 04:51:26.000938 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:51:26 dist-test-slave-1x4g krb5kdc[15648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:51:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:51:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:51:26.167812 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41281
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:38389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:41281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:26.459472 15661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:26.459735 15661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:26.459818 15661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:26.466208 15661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:51:26.466315 15661 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 04:51:26.466353 15661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:26.466403 15661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:26.466440 15661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:51:26.466490 15661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:51:26.474992 15661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:38389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:41281
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:26.476039 15661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:26.477280 15661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:26.488284 15667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:26.488363 15666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:26.488337 15669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:26.489061 15661 server_base.cc:1034] running on GCE node
I20260806 04:51:26.489701 15661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:26.490829 15661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:26.492026 15661 hybrid_clock.cc:648] HybridClock initialized: now 1785991886491970 us; error 34 us; skew 500 ppm
I20260806 04:51:26.492272 15661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:26 dist-test-slave-1x4g krb5kdc[15648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991886, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:26.495146 15661 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:51:26.496376 15661 webserver.cc:460] Webserver started at http://127.13.148.190:40271/ using document root <none> and password file <none>
I20260806 04:51:26.497087 15661 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:26.497174 15661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:26.497445 15661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:26.500289 15661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "295d1ba13dc34e5fb0be41631f1ec89f"
format_stamp: "Formatted at 2026-08-06 04:51:26 on dist-test-slave-1x4g"
server_key: "f681d5756a086b4fdda77baae04f10c1"
server_key_iv: "f245ce653498ba278c1b258b61146cea"
server_key_version: "encryptionkey@0"
I20260806 04:51:26.501091 15661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "295d1ba13dc34e5fb0be41631f1ec89f"
format_stamp: "Formatted at 2026-08-06 04:51:26 on dist-test-slave-1x4g"
server_key: "f681d5756a086b4fdda77baae04f10c1"
server_key_iv: "f245ce653498ba278c1b258b61146cea"
server_key_version: "encryptionkey@0"
I20260806 04:51:26.505862 15661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:51:26.509119 15676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:26.510330 15661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:51:26.510505 15661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "295d1ba13dc34e5fb0be41631f1ec89f"
format_stamp: "Formatted at 2026-08-06 04:51:26 on dist-test-slave-1x4g"
server_key: "f681d5756a086b4fdda77baae04f10c1"
server_key_iv: "f245ce653498ba278c1b258b61146cea"
server_key_version: "encryptionkey@0"
I20260806 04:51:26.510705 15661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:26.526319 15661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:26.530375 15661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:26.530701 15661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:26.551280 15661 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:41281
I20260806 04:51:26.551335 15739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:41281 every 8 connection(s)
I20260806 04:51:26.552546 15661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:51:26.556228 15740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:26.561928 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 15661
I20260806 04:51:26.562142 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:51:26.562759 13906 external_mini_cluster.cc:1428] Setting key dcabff5f40224165f78d5180ca653aeb
I20260806 04:51:26.566414 15740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f: Bootstrap starting.
I20260806 04:51:26.569957 15740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:26.571022 15740 log.cc:824] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:26.573824 15740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f: No bootstrap required, opened a new log
Aug 06 04:51:26 dist-test-slave-1x4g krb5kdc[15648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:26.581732 15740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295d1ba13dc34e5fb0be41631f1ec89f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41281 } }
I20260806 04:51:26.582083 15740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:26.582188 15740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 295d1ba13dc34e5fb0be41631f1ec89f, State: Initialized, Role: FOLLOWER
I20260806 04:51:26.582733 15740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295d1ba13dc34e5fb0be41631f1ec89f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41281 } }
I20260806 04:51:26.582966 15740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:51:26.583105 15740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:51:26.583271 15740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:26.584314 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:26.564605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.13.148.190:41281)
0806 04:51:26.565395 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:26.565406 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:26.565445 (+ 39us) server_negotiation.cc:407] Connection header received
0806 04:51:26.567064 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:26.567093 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:26.567475 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:26.567800 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:26.569593 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:26.570838 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:26.572330 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:26.572983 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:26.573120 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:26.577453 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:26.577462 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:26.577478 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:26.577516 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:26.580398 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:26.581145 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:26.581151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:26.581157 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:26.581301 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:26.581652 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:26.581655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:26.581657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:26.582218 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:26.582501 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:26.582937 (+ 436us) server_negotiation.cc:299] Negotiation successful
0806 04:51:26.583354 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":376,"threads_started":1}
I20260806 04:51:26.585528 15740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295d1ba13dc34e5fb0be41631f1ec89f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41281 } }
I20260806 04:51:26.585989 15740 leader_election.cc:302] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 295d1ba13dc34e5fb0be41631f1ec89f; no voters:
I20260806 04:51:26.586527 15740 leader_election.cc:288] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:51:26.586694 15745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:26.588119 15745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [term 1 LEADER]: Becoming Leader. State: Replica: 295d1ba13dc34e5fb0be41631f1ec89f, State: Running, Role: LEADER
I20260806 04:51:26.588462 15740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:51:26.588680 15745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295d1ba13dc34e5fb0be41631f1ec89f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41281 } }
I20260806 04:51:26.591616 15747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 295d1ba13dc34e5fb0be41631f1ec89f. Latest consensus state: current_term: 1 leader_uuid: "295d1ba13dc34e5fb0be41631f1ec89f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295d1ba13dc34e5fb0be41631f1ec89f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41281 } } }
I20260806 04:51:26.592336 15747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:26.591756 15746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "295d1ba13dc34e5fb0be41631f1ec89f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295d1ba13dc34e5fb0be41631f1ec89f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41281 } } }
I20260806 04:51:26.592495 15746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:26.593431 15754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:51:26.598557 15754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:51:26.606472 15754 catalog_manager.cc:1347] Generated new cluster ID: 63e29de4e5404a1eb1d5511658337624
I20260806 04:51:26.606575 15754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:51:26.618166 15754 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:51:26.619839 15754 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:51:26.631852 15754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f: Generated new TSK 0
I20260806 04:51:26.632879 15754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:51:26.682466 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41281
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:38389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:26.991207 15766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:26.991454 15766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:26.991545 15766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:26.998405 15766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:26.998508 15766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:26.998607 15766 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 04:51:26.998660 15766 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:51:26.998718 15766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:51:27.008029 15766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:38389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.190:41281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:27.009331 15766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:27.010528 15766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:27.023249 15771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:27.023303 15774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:27.023327 15772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:27.024096 15766 server_base.cc:1034] running on GCE node
I20260806 04:51:27.024758 15766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:27.025656 15766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:27.026846 15766 hybrid_clock.cc:648] HybridClock initialized: now 1785991887026799 us; error 51 us; skew 500 ppm
I20260806 04:51:27.027094 15766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:27 dist-test-slave-1x4g krb5kdc[15648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991887, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:27.030331 15766 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:51:27.031630 15766 webserver.cc:460] Webserver started at http://127.13.148.129:38615/ using document root <none> and password file <none>
I20260806 04:51:27.032233 15766 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:27.032330 15766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:27.032622 15766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:27.035498 15766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "622d92e8a8a54d7990991907b515f955"
format_stamp: "Formatted at 2026-08-06 04:51:27 on dist-test-slave-1x4g"
server_key: "6f082004e5d7dfe1119855de91a4d9d2"
server_key_iv: "555ade48ceaaa332e11bc469325525e1"
server_key_version: "encryptionkey@0"
I20260806 04:51:27.036195 15766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "622d92e8a8a54d7990991907b515f955"
format_stamp: "Formatted at 2026-08-06 04:51:27 on dist-test-slave-1x4g"
server_key: "6f082004e5d7dfe1119855de91a4d9d2"
server_key_iv: "555ade48ceaaa332e11bc469325525e1"
server_key_version: "encryptionkey@0"
I20260806 04:51:27.041087 15766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260806 04:51:27.044234 15781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:27.045414 15766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:51:27.045571 15766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "622d92e8a8a54d7990991907b515f955"
format_stamp: "Formatted at 2026-08-06 04:51:27 on dist-test-slave-1x4g"
server_key: "6f082004e5d7dfe1119855de91a4d9d2"
server_key_iv: "555ade48ceaaa332e11bc469325525e1"
server_key_version: "encryptionkey@0"
I20260806 04:51:27.045750 15766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:27.058784 15766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:27.062027 15766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:27.062292 15766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:27.064539 15766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:27.064690 15766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:27.064819 15766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:27.064930 15766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 06 04:51:27 dist-test-slave-1x4g krb5kdc[15648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991887, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:27.088166 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:27.065666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41281 (local address 127.13.148.129:54753)
0806 04:51:27.066536 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:27.066581 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:27.068554 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:27.069598 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:27.069611 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:27.070542 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:27.071507 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:27.071537 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.072906 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.072909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:27.074723 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:27.074734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.075678 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.075685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:27.075942 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:27.076723 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:27.076761 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:27.079927 (+ 3166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:27.083551 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:27.083562 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:27.083575 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:27.083989 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:27.084424 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:27.084428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:27.084432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:27.084738 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:27.085490 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:27.085495 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:27.085975 (+ 480us) client_negotiation.cc:769] Sending connection context
0806 04:51:27.086251 (+ 276us) client_negotiation.cc:240] Negotiation successful
0806 04:51:27.086705 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":390,"threads_started":1}
I20260806 04:51:27.088907 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:27.065247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:54753 (local address 127.13.148.190:41281)
0806 04:51:27.065425 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:27.065431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:27.066716 (+ 1285us) server_negotiation.cc:407] Connection header received
0806 04:51:27.068897 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:27.068903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:27.069003 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:27.069130 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:27.071787 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.072671 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.074856 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.075437 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.075567 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:27.080203 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:27.080209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:27.080213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:27.080252 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:27.083428 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:27.084152 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:27.084157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:27.084161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:27.084310 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:27.084902 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:27.084905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:27.084906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:27.085272 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:27.085422 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:27.088268 (+ 2846us) server_negotiation.cc:299] Negotiation successful
0806 04:51:27.088494 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 04:51:27.107373 15766 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:37893
I20260806 04:51:27.107447 15904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:37893 every 8 connection(s)
I20260806 04:51:27.108911 15766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:51:27.110955 15905 heartbeater.cc:347] Connected to a master server at 127.13.148.190:41281
I20260806 04:51:27.111159 15905 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:27.111887 15905 heartbeater.cc:511] Master 127.13.148.190:41281 requested a full tablet report, sending...
I20260806 04:51:27.113855 15694 ts_manager.cc:194] Registered new tserver with Master: 622d92e8a8a54d7990991907b515f955 (127.13.148.129:37893)
I20260806 04:51:27.115232 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:27.101218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41281 (local address 127.13.148.129:51133)
0806 04:51:27.101879 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:27.101901 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:27.102096 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:27.102622 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:27.102628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:27.103249 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:27.103695 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:27.103711 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.105255 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.105264 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:27.106875 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:27.106895 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.107899 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.107904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:27.108020 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:27.108666 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:27.108681 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:27.109625 (+ 944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:27.112814 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:27.112818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:27.112822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:27.113135 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:27.113586 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:27.113591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:27.113594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:27.113715 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:27.114277 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:27.114281 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:27.114460 (+ 179us) client_negotiation.cc:769] Sending connection context
0806 04:51:27.114596 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 04:51:27.114846 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":382,"threads_started":1}
I20260806 04:51:27.115242 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:27.101023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:51133 (local address 127.13.148.190:41281)
0806 04:51:27.101181 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:27.101186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:27.101982 (+ 796us) server_negotiation.cc:407] Connection header received
0806 04:51:27.102270 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:27.102275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:27.102338 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:27.102442 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:27.103880 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.105066 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.107080 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.107735 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.107857 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:27.109820 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:27.109826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:27.109830 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:27.109866 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:27.112683 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:27.113281 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:27.113287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:27.113290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:27.113485 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:27.113821 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:27.113825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:27.113828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:27.114167 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:27.114334 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:27.114694 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 04:51:27.114930 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":49}
I20260806 04:51:27.115443 15694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:54753
I20260806 04:51:27.119132 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 15766
I20260806 04:51:27.119339 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:51:27.119954 13906 external_mini_cluster.cc:1428] Setting key 45220a2ecffdf5cb3bb27ff4bb8ef3f8
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:51:27.157652 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41281
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:38389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:27.444707 15909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:27.444983 15909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:27.445158 15909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:27.451478 15909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:27.451574 15909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:27.451701 15909 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 04:51:27.451767 15909 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:51:27.451857 15909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:51:27.460009 15909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:38389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.190:41281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:27.461107 15909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:27.462272 15909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:27.473721 15914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:27.473698 15917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:27.473783 15915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:27.474339 15909 server_base.cc:1034] running on GCE node
I20260806 04:51:27.474785 15909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:27.475481 15909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:27.476680 15909 hybrid_clock.cc:648] HybridClock initialized: now 1785991887476628 us; error 26 us; skew 500 ppm
I20260806 04:51:27.477001 15909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:27 dist-test-slave-1x4g krb5kdc[15648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991887, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:27.479980 15909 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:51:27.481258 15909 webserver.cc:460] Webserver started at http://127.13.148.130:40161/ using document root <none> and password file <none>
I20260806 04:51:27.481844 15909 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:27.481995 15909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:27.482268 15909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:27.484735 15909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "c0e1b681ae3b41d991e4feb81b768e98"
format_stamp: "Formatted at 2026-08-06 04:51:27 on dist-test-slave-1x4g"
server_key: "961c3b0e849d625f19802fc5396faf1a"
server_key_iv: "0e2ed3f51ef28f217ed6d6c8eb386ee3"
server_key_version: "encryptionkey@0"
I20260806 04:51:27.485507 15909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "c0e1b681ae3b41d991e4feb81b768e98"
format_stamp: "Formatted at 2026-08-06 04:51:27 on dist-test-slave-1x4g"
server_key: "961c3b0e849d625f19802fc5396faf1a"
server_key_iv: "0e2ed3f51ef28f217ed6d6c8eb386ee3"
server_key_version: "encryptionkey@0"
I20260806 04:51:27.490324 15909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:51:27.494140 15924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:27.495380 15909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:51:27.495543 15909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "c0e1b681ae3b41d991e4feb81b768e98"
format_stamp: "Formatted at 2026-08-06 04:51:27 on dist-test-slave-1x4g"
server_key: "961c3b0e849d625f19802fc5396faf1a"
server_key_iv: "0e2ed3f51ef28f217ed6d6c8eb386ee3"
server_key_version: "encryptionkey@0"
I20260806 04:51:27.495715 15909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:27.518182 15909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:27.522219 15909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:27.522464 15909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:27.525012 15909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:27.525146 15909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:27.525269 15909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:27.525388 15909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 06 04:51:27 dist-test-slave-1x4g krb5kdc[15648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991887, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:27.545899 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:27.525353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:55951 (local address 127.13.148.190:41281)
0806 04:51:27.525539 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:27.525543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:27.527019 (+ 1476us) server_negotiation.cc:407] Connection header received
0806 04:51:27.528988 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:27.528993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:27.529069 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:27.529176 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:27.531093 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.531825 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.534051 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.534645 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.534772 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:27.539403 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:27.539409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:27.539414 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:27.539462 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:27.542416 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:27.543468 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:27.543474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:27.543480 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:27.543638 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:27.544159 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:27.544164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:27.544167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:27.544453 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:27.544612 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:27.545400 (+ 788us) server_negotiation.cc:299] Negotiation successful
0806 04:51:27.545610 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:51:27.547101 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:27.525993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41281 (local address 127.13.148.130:55951)
0806 04:51:27.526870 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:27.526906 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:27.528688 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:27.529370 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:27.529377 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:27.529994 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:27.530900 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:27.530924 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.531974 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.531981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:27.533902 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:27.533913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.534888 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.534896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:27.535086 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:27.535751 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:27.535771 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:27.539156 (+ 3385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:27.542578 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:27.542588 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:27.542609 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:27.543330 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:27.543748 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:27.543752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:27.543755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:27.544034 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:27.544585 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:27.544592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:27.544990 (+ 398us) client_negotiation.cc:769] Sending connection context
0806 04:51:27.545307 (+ 317us) client_negotiation.cc:240] Negotiation successful
0806 04:51:27.545742 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":335,"threads_started":1}
I20260806 04:51:27.575677 15909 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:36263
I20260806 04:51:27.575757 16045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:36263 every 8 connection(s)
I20260806 04:51:27.577073 15909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:51:27.579299 16046 heartbeater.cc:347] Connected to a master server at 127.13.148.190:41281
I20260806 04:51:27.579519 16046 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:27.580188 16046 heartbeater.cc:511] Master 127.13.148.190:41281 requested a full tablet report, sending...
I20260806 04:51:27.580646 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:27.567830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:59181 (local address 127.13.148.190:41281)
0806 04:51:27.568068 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:27.568076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:27.568795 (+ 719us) server_negotiation.cc:407] Connection header received
0806 04:51:27.569020 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:27.569026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:27.569111 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:27.569228 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:27.570497 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.571366 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.572647 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.573363 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.573523 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:27.575406 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:27.575412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:27.575416 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:27.575451 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:27.578232 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:27.578922 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:27.578926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:27.578927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:27.579061 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:27.579433 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:27.579439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:27.579442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:27.579743 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:27.579842 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:27.580283 (+ 441us) server_negotiation.cc:299] Negotiation successful
0806 04:51:27.580448 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 04:51:27.580914 16027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:27.567837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41281 (local address 127.13.148.130:59181)
0806 04:51:27.568629 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:27.568665 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:27.568873 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:27.569253 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:27.569257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:27.569906 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:27.570353 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:27.570367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.571495 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.571500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:27.572480 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:27.572490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.573586 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.573593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:27.573799 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:27.574415 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:27.574430 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:27.575250 (+ 820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:27.578391 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:27.578396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:27.578401 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:27.578825 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:27.579169 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:27.579173 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:27.579177 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:27.579323 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:27.579853 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:27.579857 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:27.580036 (+ 179us) client_negotiation.cc:769] Sending connection context
0806 04:51:27.580198 (+ 162us) client_negotiation.cc:240] Negotiation successful
0806 04:51:27.580474 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":401,"threads_started":1}
I20260806 04:51:27.581624 15693 ts_manager.cc:194] Registered new tserver with Master: c0e1b681ae3b41d991e4feb81b768e98 (127.13.148.130:36263)
I20260806 04:51:27.582482 15693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:55951
I20260806 04:51:27.583012 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 15909
I20260806 04:51:27.583171 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:51:27.583659 13906 external_mini_cluster.cc:1428] Setting key bc361124aeb7487533aa05ef13458530
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
I20260806 04:51:27.623517 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41281
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:38389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:27.915176 16050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:27.915387 16050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:27.915557 16050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:27.922113 16050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:27.922194 16050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:27.922262 16050 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 04:51:27.922323 16050 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:51:27.922437 16050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:51:27.932019 16050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:38389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.190:41281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:27.933197 16050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:27.934392 16050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:27.946220 16059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:27.946254 16057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:27.946434 16056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:27.947009 16050 server_base.cc:1034] running on GCE node
I20260806 04:51:27.947496 16050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:27.948380 16050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:27.949569 16050 hybrid_clock.cc:648] HybridClock initialized: now 1785991887949537 us; error 29 us; skew 500 ppm
I20260806 04:51:27.949848 16050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:27 dist-test-slave-1x4g krb5kdc[15648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991887, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:27.952641 16050 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:51:27.953796 16050 webserver.cc:460] Webserver started at http://127.13.148.131:37019/ using document root <none> and password file <none>
I20260806 04:51:27.954357 16050 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:27.954475 16050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:27.954756 16050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:27.957327 16050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "3f5f9dd993d04dc29ba74dfab215088d"
format_stamp: "Formatted at 2026-08-06 04:51:27 on dist-test-slave-1x4g"
server_key: "6dc1dabd201d8124a0a894030af93ca1"
server_key_iv: "1a810335a8a3e824c6e67b01397f80f1"
server_key_version: "encryptionkey@0"
I20260806 04:51:27.958097 16050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "3f5f9dd993d04dc29ba74dfab215088d"
format_stamp: "Formatted at 2026-08-06 04:51:27 on dist-test-slave-1x4g"
server_key: "6dc1dabd201d8124a0a894030af93ca1"
server_key_iv: "1a810335a8a3e824c6e67b01397f80f1"
server_key_version: "encryptionkey@0"
I20260806 04:51:27.962519 16050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:51:27.965497 16066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:27.966572 16050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:51:27.966724 16050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "3f5f9dd993d04dc29ba74dfab215088d"
format_stamp: "Formatted at 2026-08-06 04:51:27 on dist-test-slave-1x4g"
server_key: "6dc1dabd201d8124a0a894030af93ca1"
server_key_iv: "1a810335a8a3e824c6e67b01397f80f1"
server_key_version: "encryptionkey@0"
I20260806 04:51:27.966887 16050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:27.981830 16050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:27.984930 16050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:27.985183 16050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:27.987301 16050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:27.987422 16050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:27.987563 16050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:27.987656 16050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:51:27 dist-test-slave-1x4g krb5kdc[15648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991887, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:28.007934 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:27.987964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:37035 (local address 127.13.148.190:41281)
0806 04:51:27.988148 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:27.988154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:27.989525 (+ 1371us) server_negotiation.cc:407] Connection header received
0806 04:51:27.991648 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:27.991654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:27.991747 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:27.991822 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:27.994146 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.995277 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.996672 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.997498 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.997649 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.001507 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:28.001512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:28.001515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:28.001544 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:28.004698 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:28.005565 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:28.005570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:28.005573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:28.005699 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:28.006311 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:28.006317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:28.006320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:28.006616 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:28.006769 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:28.007338 (+ 569us) server_negotiation.cc:299] Negotiation successful
0806 04:51:28.007596 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 04:51:28.008996 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:27.988467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41281 (local address 127.13.148.131:37035)
0806 04:51:27.989355 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:27.989401 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:27.991320 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:27.992320 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:27.992329 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:27.993125 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:27.993915 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:27.993935 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.995424 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.995429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:27.996523 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:27.996534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:27.997721 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:27.997726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:27.997903 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:27.998663 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:27.998710 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:28.001239 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:28.004833 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:28.004841 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:28.004854 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:28.005462 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:28.005804 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:28.005808 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:28.005811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:28.006128 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:28.006736 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:28.006744 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:28.007033 (+ 289us) client_negotiation.cc:769] Sending connection context
0806 04:51:28.007249 (+ 216us) client_negotiation.cc:240] Negotiation successful
0806 04:51:28.007735 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":403,"threads_started":1}
I20260806 04:51:28.032541 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.015802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:33969 (local address 127.13.148.190:41281)
0806 04:51:28.015989 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:28.015996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:28.016746 (+ 750us) server_negotiation.cc:407] Connection header received
0806 04:51:28.016991 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:28.016998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:28.017092 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:28.017193 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:28.019007 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.020048 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.022402 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.023142 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.023257 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.026056 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:28.026059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:28.026062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:28.026088 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:28.028834 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:28.029862 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:28.029869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:28.029872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:28.030072 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:28.030616 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:28.030622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:28.030626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:28.031001 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:28.031160 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:28.031818 (+ 658us) server_negotiation.cc:299] Negotiation successful
0806 04:51:28.032046 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:51:28.032979 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.015978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41281 (local address 127.13.148.131:33969)
0806 04:51:28.016605 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:28.016628 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:28.016836 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:28.017402 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:28.017415 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:28.018196 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:28.018794 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.018814 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.020201 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.020209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.022207 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.022227 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.023690 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.023703 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.023922 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.024711 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:28.024745 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:28.025896 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:28.028973 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:28.028981 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:28.028987 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:28.029716 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:28.030227 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:28.030234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:28.030239 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:28.030424 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:28.031140 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:28.031147 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:28.031469 (+ 322us) client_negotiation.cc:769] Sending connection context
0806 04:51:28.031667 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 04:51:28.032070 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":352,"threads_started":1}
I20260806 04:51:28.035574 16050 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:39843
I20260806 04:51:28.035661 16188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:39843 every 8 connection(s)
I20260806 04:51:28.036931 16050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:51:28.038894 16190 heartbeater.cc:347] Connected to a master server at 127.13.148.190:41281
I20260806 04:51:28.039490 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 16050
I20260806 04:51:28.039690 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:51:28.040138 13906 external_mini_cluster.cc:1428] Setting key 47ebf0970a37ab0e8a82be2920d3168b
I20260806 04:51:28.040670 16190 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:28.041356 16190 heartbeater.cc:511] Master 127.13.148.190:41281 requested a full tablet report, sending...
I20260806 04:51:28.042706 15694 ts_manager.cc:194] Registered new tserver with Master: 3f5f9dd993d04dc29ba74dfab215088d (127.13.148.131:39843)
I20260806 04:51:28.043411 15694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:37035
I20260806 04:51:28.054539 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 04:51:28 dist-test-slave-1x4g krb5kdc[15648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 04:51:28 dist-test-slave-1x4g krb5kdc[15648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:28.097023 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.084747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.13.148.190:41281)
0806 04:51:28.084929 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:28.084933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:28.085281 (+ 348us) server_negotiation.cc:407] Connection header received
0806 04:51:28.085419 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:28.085422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:28.085484 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:28.085563 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:28.086725 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.087328 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.088558 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.089088 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.089174 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.092737 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:28.092741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:28.092744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:28.092769 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:28.094910 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:28.095402 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:28.095405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:28.095407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:28.095526 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:28.095848 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:28.095851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:28.095853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:28.096104 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:28.096272 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:28.096573 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 04:51:28.096749 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 04:51:28.100828 15694 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:51:28.107105 15694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:51:28.118541 15905 heartbeater.cc:503] Master 127.13.148.190:41281 was elected leader, sending a full tablet report...
I20260806 04:51:28.129220 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.118434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:37893 (local address 127.0.0.1:34076)
0806 04:51:28.120898 (+ 2464us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:28.120935 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:28.121174 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:28.122663 (+ 1489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:28.122667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:28.122691 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:28.123109 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.123125 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.124842 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.124847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.126424 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.126436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.128164 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.128170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.128355 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.128412 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 04:51:28.128554 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 04:51:28.128756 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2219,"mutex_wait_us":48,"thread_start_us":2860,"threads_started":1}
I20260806 04:51:28.129395 16205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.118736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34076 (local address 127.13.148.129:37893)
0806 04:51:28.120347 (+ 1611us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:28.120358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:28.121033 (+ 675us) server_negotiation.cc:407] Connection header received
0806 04:51:28.121451 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:28.121459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:28.121680 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:28.121816 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:28.123277 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.124660 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.126564 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.127971 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.128155 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.128322 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:28.128897 (+ 575us) server_negotiation.cc:299] Negotiation successful
0806 04:51:28.129079 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1424,"thread_start_us":1295,"threads_started":1}
I20260806 04:51:28.129988 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.119700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41976 (local address 127.13.148.131:39843)
0806 04:51:28.120867 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:28.120877 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:28.121621 (+ 744us) server_negotiation.cc:407] Connection header received
0806 04:51:28.121721 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:28.121727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:28.121931 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:28.122125 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:28.123009 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.124474 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.126998 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.128407 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.128605 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.128778 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:28.129571 (+ 793us) server_negotiation.cc:299] Negotiation successful
0806 04:51:28.129712 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":970,"thread_start_us":829,"threads_started":1}
I20260806 04:51:28.130288 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.121962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:36263 (local address 127.0.0.1:38924)
0806 04:51:28.122407 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:28.122420 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:28.122543 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:28.123180 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:28.123186 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:28.123208 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:28.123564 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.123575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.125241 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.125248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.127256 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.127269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.129543 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.129555 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.129714 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.129739 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 04:51:28.129825 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 04:51:28.129922 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":1,"spinlock_wait_cycles":17152,"thread_start_us":224,"threads_started":1}
I20260806 04:51:28.130326 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.119062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:39843 (local address 127.0.0.1:41976)
0806 04:51:28.121342 (+ 2280us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:28.121359 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:28.121471 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:28.122405 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:28.122411 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:28.122442 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:28.122834 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.122843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.124610 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.124617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.126684 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.126698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.128628 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.128634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.128813 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.128840 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 04:51:28.128939 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 04:51:28.129049 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2126,"thread_start_us":2469,"threads_started":1}
I20260806 04:51:28.130720 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.122214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.13.148.130:36263)
0806 04:51:28.122683 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:28.122689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:28.122710 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:51:28.122796 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:28.122803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:28.122998 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:28.123177 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:28.123821 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.125114 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.127432 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.128830 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.129000 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.129148 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:28.129893 (+ 745us) server_negotiation.cc:299] Negotiation successful
0806 04:51:28.130197 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":248,"threads_started":1}
I20260806 04:51:28.132891 15834 tablet_service.cc:1467] Processing CreateTablet for tablet d41c1e82dfce48c192e3be49b5a492c7 (DEFAULT_TABLE table=test-table [id=88bd669b94e74a809365b8d0b90fcc2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:28.133556 15976 tablet_service.cc:1467] Processing CreateTablet for tablet d41c1e82dfce48c192e3be49b5a492c7 (DEFAULT_TABLE table=test-table [id=88bd669b94e74a809365b8d0b90fcc2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:28.133559 16118 tablet_service.cc:1467] Processing CreateTablet for tablet d41c1e82dfce48c192e3be49b5a492c7 (DEFAULT_TABLE table=test-table [id=88bd669b94e74a809365b8d0b90fcc2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:28.134230 15834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d41c1e82dfce48c192e3be49b5a492c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:28.134575 15976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d41c1e82dfce48c192e3be49b5a492c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:28.134794 16118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d41c1e82dfce48c192e3be49b5a492c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:28.144724 16211 tablet_bootstrap.cc:492] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98: Bootstrap starting.
I20260806 04:51:28.147382 16213 tablet_bootstrap.cc:492] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d: Bootstrap starting.
I20260806 04:51:28.147748 16211 tablet_bootstrap.cc:654] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:28.149283 16211 log.cc:824] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:28.149592 16212 tablet_bootstrap.cc:492] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955: Bootstrap starting.
I20260806 04:51:28.153604 16212 tablet_bootstrap.cc:654] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:28.154407 16213 tablet_bootstrap.cc:654] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:28.154783 16212 log.cc:824] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:28.155486 16213 log.cc:824] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:28.156229 16211 tablet_bootstrap.cc:492] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98: No bootstrap required, opened a new log
I20260806 04:51:28.156490 16211 ts_tablet_manager.cc:1397] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260806 04:51:28.157599 16212 tablet_bootstrap.cc:492] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955: No bootstrap required, opened a new log
I20260806 04:51:28.157855 16212 ts_tablet_manager.cc:1397] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260806 04:51:28.158146 16213 tablet_bootstrap.cc:492] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d: No bootstrap required, opened a new log
I20260806 04:51:28.158399 16213 ts_tablet_manager.cc:1397] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20260806 04:51:28.164268 16213 raft_consensus.cc:348] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } }
I20260806 04:51:28.164597 16213 raft_consensus.cc:374] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:28.164721 16213 raft_consensus.cc:729] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f5f9dd993d04dc29ba74dfab215088d, State: Initialized, Role: FOLLOWER
I20260806 04:51:28.165329 16213 consensus_queue.cc:260] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } }
I20260806 04:51:28.166324 16190 heartbeater.cc:503] Master 127.13.148.190:41281 was elected leader, sending a full tablet report...
I20260806 04:51:28.166755 16213 ts_tablet_manager.cc:1428] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d: Time spent starting tablet: real 0.008s user 0.003s sys 0.006s
I20260806 04:51:28.166921 16211 raft_consensus.cc:348] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } }
I20260806 04:51:28.167282 16211 raft_consensus.cc:374] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:28.167126 16212 raft_consensus.cc:348] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } }
I20260806 04:51:28.167383 16211 raft_consensus.cc:729] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e1b681ae3b41d991e4feb81b768e98, State: Initialized, Role: FOLLOWER
I20260806 04:51:28.167477 16212 raft_consensus.cc:374] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:28.167620 16212 raft_consensus.cc:729] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622d92e8a8a54d7990991907b515f955, State: Initialized, Role: FOLLOWER
I20260806 04:51:28.167997 16211 consensus_queue.cc:260] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } }
I20260806 04:51:28.168190 16212 consensus_queue.cc:260] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } }
I20260806 04:51:28.170607 16212 ts_tablet_manager.cc:1428] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260806 04:51:28.171622 16046 heartbeater.cc:503] Master 127.13.148.190:41281 was elected leader, sending a full tablet report...
I20260806 04:51:28.172163 16211 ts_tablet_manager.cc:1428] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260806 04:51:28.218231 16217 raft_consensus.cc:482] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:51:28.218562 16217 raft_consensus.cc:504] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } }
I20260806 04:51:28.220112 16217 leader_election.cc:288] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 622d92e8a8a54d7990991907b515f955 (127.13.148.129:37893), c0e1b681ae3b41d991e4feb81b768e98 (127.13.148.130:36263)
I20260806 04:51:28.225152 16205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.220549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:33849 (local address 127.13.148.129:37893)
0806 04:51:28.220748 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:28.220754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:28.220798 (+ 44us) server_negotiation.cc:407] Connection header received
0806 04:51:28.220875 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:28.220882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:28.221004 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:28.221109 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:28.221558 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.222186 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.223582 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.224297 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.224467 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.224533 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:28.224841 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 04:51:28.224959 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 04:51:28.225157 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.220384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:37893 (local address 127.13.148.131:33849)
0806 04:51:28.220580 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:28.220600 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:28.220719 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:28.221243 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:28.221246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:28.221265 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:28.221427 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.221433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.222325 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.222329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.223420 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.223431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.224463 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.224468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.224646 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.224668 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:51:28.224788 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 04:51:28.224917 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 04:51:28.225848 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.220718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:36263 (local address 127.13.148.131:34023)
0806 04:51:28.221331 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:28.221347 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:28.221478 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:28.221760 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:28.221764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:28.221783 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:28.222063 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.222070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.222934 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.222946 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.224128 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.224139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.225244 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.225249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.225398 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.225437 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 04:51:28.225521 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 04:51:28.225627 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":331,"threads_started":1}
I20260806 04:51:28.225971 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.220987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:34023 (local address 127.13.148.130:36263)
0806 04:51:28.221187 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:28.221193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:28.221427 (+ 234us) server_negotiation.cc:407] Connection header received
0806 04:51:28.221602 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:28.221612 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:28.221664 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:28.221748 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:28.222191 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.222816 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.224268 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.225051 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.225211 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.225274 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:28.225579 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 04:51:28.225700 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 04:51:28.226192 15854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d41c1e82dfce48c192e3be49b5a492c7" candidate_uuid: "3f5f9dd993d04dc29ba74dfab215088d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622d92e8a8a54d7990991907b515f955" is_pre_election: true
I20260806 04:51:28.226666 15854 raft_consensus.cc:2454] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f5f9dd993d04dc29ba74dfab215088d in term 0.
I20260806 04:51:28.226869 15996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d41c1e82dfce48c192e3be49b5a492c7" candidate_uuid: "3f5f9dd993d04dc29ba74dfab215088d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e1b681ae3b41d991e4feb81b768e98" is_pre_election: true
I20260806 04:51:28.227326 16069 leader_election.cc:302] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f5f9dd993d04dc29ba74dfab215088d, 622d92e8a8a54d7990991907b515f955; no voters:
I20260806 04:51:28.227396 15996 raft_consensus.cc:2454] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f5f9dd993d04dc29ba74dfab215088d in term 0.
I20260806 04:51:28.227774 16217 raft_consensus.cc:2781] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:51:28.227938 16217 raft_consensus.cc:482] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:51:28.228073 16217 raft_consensus.cc:3037] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:28.230224 16217 raft_consensus.cc:504] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } }
I20260806 04:51:28.230840 16217 leader_election.cc:288] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [CANDIDATE]: Term 1 election: Requested vote from peers 622d92e8a8a54d7990991907b515f955 (127.13.148.129:37893), c0e1b681ae3b41d991e4feb81b768e98 (127.13.148.130:36263)
I20260806 04:51:28.231166 15854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d41c1e82dfce48c192e3be49b5a492c7" candidate_uuid: "3f5f9dd993d04dc29ba74dfab215088d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622d92e8a8a54d7990991907b515f955"
I20260806 04:51:28.231235 15996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d41c1e82dfce48c192e3be49b5a492c7" candidate_uuid: "3f5f9dd993d04dc29ba74dfab215088d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e1b681ae3b41d991e4feb81b768e98"
I20260806 04:51:28.231359 15854 raft_consensus.cc:3037] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:28.231446 15996 raft_consensus.cc:3037] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:28.233551 15854 raft_consensus.cc:2454] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f5f9dd993d04dc29ba74dfab215088d in term 1.
I20260806 04:51:28.233615 15996 raft_consensus.cc:2454] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f5f9dd993d04dc29ba74dfab215088d in term 1.
I20260806 04:51:28.233950 16069 leader_election.cc:302] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f5f9dd993d04dc29ba74dfab215088d, 622d92e8a8a54d7990991907b515f955; no voters:
I20260806 04:51:28.234226 16217 raft_consensus.cc:2781] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:28.234619 16217 raft_consensus.cc:686] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [term 1 LEADER]: Becoming Leader. State: Replica: 3f5f9dd993d04dc29ba74dfab215088d, State: Running, Role: LEADER
I20260806 04:51:28.235078 16217 consensus_queue.cc:237] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } }
I20260806 04:51:28.239218 15693 catalog_manager.cc:5495] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f5f9dd993d04dc29ba74dfab215088d (127.13.148.131). New cstate: current_term: 1 leader_uuid: "3f5f9dd993d04dc29ba74dfab215088d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } health_report { overall_health: UNKNOWN } } }
W20260806 04:51:28.288460 16191 tablet.cc:2369] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 04:51:28 dist-test-slave-1x4g krb5kdc[15648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991886, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:28.311326 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.298912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.13.148.190:41281)
0806 04:51:28.299099 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:28.299105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:28.299580 (+ 475us) server_negotiation.cc:407] Connection header received
0806 04:51:28.299750 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:28.299754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:28.299817 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:28.299889 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:28.301000 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.301720 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.302828 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.303243 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.303361 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.306426 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:28.306430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:28.306433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:28.306462 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:28.308668 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:28.309314 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:28.309320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:28.309324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:28.309464 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:28.309837 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:28.309841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:28.309844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:28.310097 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:28.310241 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:28.310602 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 04:51:28.310961 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":50}
I20260806 04:51:28.311350 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.298982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41281 (local address 127.0.0.1:58266)
0806 04:51:28.299494 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:28.299507 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:28.299645 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:28.299908 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:28.299911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:28.300594 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:28.300869 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.300880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.301832 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.301838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.302709 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.302719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.303394 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.303398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.303563 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.304209 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:28.304224 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:28.306268 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:28.308821 (+ 2553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:28.308833 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:28.308841 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:28.309207 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:28.309575 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:28.309581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:28.309584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:28.309735 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:28.310214 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:28.310223 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:28.310364 (+ 141us) client_negotiation.cc:769] Sending connection context
0806 04:51:28.310508 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 04:51:28.310804 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":267,"threads_started":1}
I20260806 04:51:28.315940 15693 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:58266:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 04:51:28.316960 15693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:51:28.323040 15976 tablet_service.cc:1467] Processing CreateTablet for tablet dbffce1979aa4d889895df866b0ab9fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03718ed54dc0437aa122178ee1e77816]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:51:28.322933 15834 tablet_service.cc:1467] Processing CreateTablet for tablet dbffce1979aa4d889895df866b0ab9fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03718ed54dc0437aa122178ee1e77816]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:51:28.323146 16118 tablet_service.cc:1467] Processing CreateTablet for tablet dbffce1979aa4d889895df866b0ab9fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03718ed54dc0437aa122178ee1e77816]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:51:28.323923 15976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbffce1979aa4d889895df866b0ab9fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:28.323925 15834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbffce1979aa4d889895df866b0ab9fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:28.324061 16118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbffce1979aa4d889895df866b0ab9fd. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 04:51:28.329043 16047 tablet.cc:2369] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:51:28.330461 16213 tablet_bootstrap.cc:492] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d: Bootstrap starting.
I20260806 04:51:28.330572 16211 tablet_bootstrap.cc:492] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98: Bootstrap starting.
I20260806 04:51:28.331734 16213 tablet_bootstrap.cc:654] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:28.331873 16211 tablet_bootstrap.cc:654] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:28.332065 16212 tablet_bootstrap.cc:492] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955: Bootstrap starting.
I20260806 04:51:28.333613 16212 tablet_bootstrap.cc:654] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:28.334041 16213 tablet_bootstrap.cc:492] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d: No bootstrap required, opened a new log
I20260806 04:51:28.334187 16213 ts_tablet_manager.cc:1397] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260806 04:51:28.335062 16213 raft_consensus.cc:348] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.335393 16213 raft_consensus.cc:374] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:28.335507 16213 raft_consensus.cc:729] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f5f9dd993d04dc29ba74dfab215088d, State: Initialized, Role: FOLLOWER
I20260806 04:51:28.335754 16213 consensus_queue.cc:260] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.336200 16217 tablet_replica.cc:439] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } }
I20260806 04:51:28.336499 16217 tablet_replica.cc:442] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:51:28.336364 16213 ts_tablet_manager.cc:1428] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260806 04:51:28.338361 16212 tablet_bootstrap.cc:492] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955: No bootstrap required, opened a new log
I20260806 04:51:28.338527 16212 ts_tablet_manager.cc:1397] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 04:51:28.338737 16211 tablet_bootstrap.cc:492] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98: No bootstrap required, opened a new log
I20260806 04:51:28.338869 16211 ts_tablet_manager.cc:1397] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260806 04:51:28.339198 16212 raft_consensus.cc:348] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.339375 16212 raft_consensus.cc:374] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:28.339491 16212 raft_consensus.cc:729] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622d92e8a8a54d7990991907b515f955, State: Initialized, Role: FOLLOWER
I20260806 04:51:28.339486 16211 raft_consensus.cc:348] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.339730 16211 raft_consensus.cc:374] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:28.339818 16211 raft_consensus.cc:729] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e1b681ae3b41d991e4feb81b768e98, State: Initialized, Role: FOLLOWER
I20260806 04:51:28.339710 16212 consensus_queue.cc:260] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.340021 16211 consensus_queue.cc:260] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.340173 16212 ts_tablet_manager.cc:1428] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 04:51:28.340425 16218 tablet_replica.cc:439] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } }
I20260806 04:51:28.340476 16219 tablet_replica.cc:439] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } }
I20260806 04:51:28.340736 16218 tablet_replica.cc:442] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:51:28.340812 16219 tablet_replica.cc:442] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:51:28.341027 16211 ts_tablet_manager.cc:1428] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260806 04:51:28.361099 15906 tablet.cc:2369] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:51:28.396209 16217 raft_consensus.cc:482] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:51:28.396407 16217 raft_consensus.cc:504] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.397048 16217 leader_election.cc:288] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 622d92e8a8a54d7990991907b515f955 (127.13.148.129:37893), c0e1b681ae3b41d991e4feb81b768e98 (127.13.148.130:36263)
I20260806 04:51:28.397567 15854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbffce1979aa4d889895df866b0ab9fd" candidate_uuid: "3f5f9dd993d04dc29ba74dfab215088d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622d92e8a8a54d7990991907b515f955" is_pre_election: true
I20260806 04:51:28.397619 15996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbffce1979aa4d889895df866b0ab9fd" candidate_uuid: "3f5f9dd993d04dc29ba74dfab215088d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e1b681ae3b41d991e4feb81b768e98" is_pre_election: true
I20260806 04:51:28.397845 15854 raft_consensus.cc:2454] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f5f9dd993d04dc29ba74dfab215088d in term 0.
I20260806 04:51:28.397893 15996 raft_consensus.cc:2454] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f5f9dd993d04dc29ba74dfab215088d in term 0.
I20260806 04:51:28.398283 16069 leader_election.cc:302] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f5f9dd993d04dc29ba74dfab215088d, 622d92e8a8a54d7990991907b515f955; no voters:
I20260806 04:51:28.398627 16217 raft_consensus.cc:2781] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:51:28.398757 16217 raft_consensus.cc:482] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:51:28.398851 16217 raft_consensus.cc:3037] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:28.400804 16217 raft_consensus.cc:504] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.401312 16217 leader_election.cc:288] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [CANDIDATE]: Term 1 election: Requested vote from peers 622d92e8a8a54d7990991907b515f955 (127.13.148.129:37893), c0e1b681ae3b41d991e4feb81b768e98 (127.13.148.130:36263)
I20260806 04:51:28.401723 15854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbffce1979aa4d889895df866b0ab9fd" candidate_uuid: "3f5f9dd993d04dc29ba74dfab215088d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622d92e8a8a54d7990991907b515f955"
I20260806 04:51:28.401829 15996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbffce1979aa4d889895df866b0ab9fd" candidate_uuid: "3f5f9dd993d04dc29ba74dfab215088d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e1b681ae3b41d991e4feb81b768e98"
I20260806 04:51:28.401919 15854 raft_consensus.cc:3037] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:28.402048 15996 raft_consensus.cc:3037] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:28.404474 15854 raft_consensus.cc:2454] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f5f9dd993d04dc29ba74dfab215088d in term 1.
I20260806 04:51:28.404525 15996 raft_consensus.cc:2454] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f5f9dd993d04dc29ba74dfab215088d in term 1.
I20260806 04:51:28.404892 16069 leader_election.cc:302] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f5f9dd993d04dc29ba74dfab215088d, c0e1b681ae3b41d991e4feb81b768e98; no voters:
I20260806 04:51:28.405107 16217 raft_consensus.cc:2781] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:28.405299 16217 raft_consensus.cc:686] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [term 1 LEADER]: Becoming Leader. State: Replica: 3f5f9dd993d04dc29ba74dfab215088d, State: Running, Role: LEADER
I20260806 04:51:28.405555 16217 consensus_queue.cc:237] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.406828 16221 tablet_replica.cc:439] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d: TxnStatusTablet state changed. Reason: New leader 3f5f9dd993d04dc29ba74dfab215088d. Latest consensus state: current_term: 1 leader_uuid: "3f5f9dd993d04dc29ba74dfab215088d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } }
I20260806 04:51:28.407109 16221 tablet_replica.cc:442] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:51:28.407833 16233 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 04:51:28.408109 16233 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 04:51:28.408116 15693 catalog_manager.cc:5495] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f5f9dd993d04dc29ba74dfab215088d (127.13.148.131). New cstate: current_term: 1 leader_uuid: "3f5f9dd993d04dc29ba74dfab215088d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } health_report { overall_health: HEALTHY } } }
I20260806 04:51:28.436566 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.430971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:39843 (local address 127.0.0.1:41984)
0806 04:51:28.431249 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:28.431264 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:28.431353 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:28.431805 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:28.431808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:28.431817 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:51:28.431958 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.431965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.433233 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.433236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.434166 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.434178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.435132 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.435136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.435293 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.435349 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:51:28.436216 (+ 867us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:51:28.436229 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 04:51:28.436274 (+ 45us) client_negotiation.cc:240] Negotiation successful
0806 04:51:28.436365 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260806 04:51:28.436638 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.431095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41984 (local address 127.13.148.131:39843)
0806 04:51:28.431277 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:28.431283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:28.431303 (+ 20us) server_negotiation.cc:407] Connection header received
0806 04:51:28.431548 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:28.431553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:28.431643 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:28.431778 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:51:28.432099 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.433126 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.434296 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.434990 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.435116 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.435544 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:51:28.436122 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:51:28.436262 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:28.436304 (+ 42us) server_negotiation.cc:299] Negotiation successful
0806 04:51:28.436381 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:51:28.443537 15996 raft_consensus.cc:1261] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98 [term 1 FOLLOWER]: Refusing update from remote peer 3f5f9dd993d04dc29ba74dfab215088d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:28.443635 15854 raft_consensus.cc:1261] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955 [term 1 FOLLOWER]: Refusing update from remote peer 3f5f9dd993d04dc29ba74dfab215088d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:28.444470 16221 consensus_queue.cc:1035] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:28.445097 16217 consensus_queue.cc:1035] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d [LEADER]: Connected to new peer: Peer: permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:28.454641 16219 tablet_replica.cc:439] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955: TxnStatusTablet state changed. Reason: New leader 3f5f9dd993d04dc29ba74dfab215088d. Latest consensus state: current_term: 1 leader_uuid: "3f5f9dd993d04dc29ba74dfab215088d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } }
I20260806 04:51:28.454917 16219 tablet_replica.cc:442] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:51:28.457484 16236 mvcc.cc:205] Tried to move back new op lower bound from 7315422775056138240 to 7315422774910930944. Current Snapshot: MvccSnapshot[applied={T|T < 7315422775056138240}]
I20260806 04:51:28.458607 16237 mvcc.cc:205] Tried to move back new op lower bound from 7315422775056138240 to 7315422774910930944. Current Snapshot: MvccSnapshot[applied={T|T < 7315422775056138240}]
I20260806 04:51:28.459506 16219 tablet_replica.cc:439] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f5f9dd993d04dc29ba74dfab215088d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } }
I20260806 04:51:28.459757 16219 tablet_replica.cc:442] T dbffce1979aa4d889895df866b0ab9fd P 622d92e8a8a54d7990991907b515f955: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:51:28.460453 16218 tablet_replica.cc:439] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98: TxnStatusTablet state changed. Reason: New leader 3f5f9dd993d04dc29ba74dfab215088d. Latest consensus state: current_term: 1 leader_uuid: "3f5f9dd993d04dc29ba74dfab215088d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } }
I20260806 04:51:28.460716 16218 tablet_replica.cc:442] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:51:28.463275 16217 tablet_replica.cc:439] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f5f9dd993d04dc29ba74dfab215088d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } }
I20260806 04:51:28.463577 16217 tablet_replica.cc:442] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:51:28.466091 16221 tablet_replica.cc:439] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f5f9dd993d04dc29ba74dfab215088d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } }
I20260806 04:51:28.466332 16221 tablet_replica.cc:442] T dbffce1979aa4d889895df866b0ab9fd P 3f5f9dd993d04dc29ba74dfab215088d: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:51:28.470080 16238 mvcc.cc:205] Tried to move back new op lower bound from 7315422775056138240 to 7315422774910930944. Current Snapshot: MvccSnapshot[applied={T|T < 7315422775056138240}]
I20260806 04:51:28.474319 16218 tablet_replica.cc:439] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f5f9dd993d04dc29ba74dfab215088d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } } }
I20260806 04:51:28.474570 16218 tablet_replica.cc:442] T dbffce1979aa4d889895df866b0ab9fd P c0e1b681ae3b41d991e4feb81b768e98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:51:28.483240 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.478127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42000 (local address 127.13.148.131:39843)
0806 04:51:28.478314 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:28.478320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:28.478340 (+ 20us) server_negotiation.cc:407] Connection header received
0806 04:51:28.478400 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:28.478405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:28.478502 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:28.478597 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:51:28.478927 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.480183 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.481466 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.482019 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.482148 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.482573 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:51:28.482693 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:51:28.482770 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:28.482903 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 04:51:28.482973 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:51:28.496874 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.492573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:46681 (local address 127.13.148.131:39843)
0806 04:51:28.492712 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:28.492717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:28.492734 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:51:28.492929 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:28.492937 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:28.493003 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:28.493090 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:51:28.493373 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.494007 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.495195 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.495568 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.495690 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.496090 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:51:28.496179 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:51:28.496265 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:28.496394 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 04:51:28.496455 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":52}
I20260806 04:51:28.496873 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.492509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:39843 (local address 127.13.148.131:46681)
0806 04:51:28.492683 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:28.492697 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:28.492817 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:28.493122 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:28.493124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:28.493133 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:51:28.493268 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.493273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.494116 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.494123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.495072 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.495083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.495698 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.495703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.495852 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.495893 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:51:28.496265 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:51:28.496278 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 04:51:28.496377 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 04:51:28.496498 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":3}
I20260806 04:51:28.505124 15996 raft_consensus.cc:1261] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 FOLLOWER]: Refusing update from remote peer 3f5f9dd993d04dc29ba74dfab215088d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:28.505275 15854 raft_consensus.cc:1261] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955 [term 1 FOLLOWER]: Refusing update from remote peer 3f5f9dd993d04dc29ba74dfab215088d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:28.505651 16217 consensus_queue.cc:1035] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:28.505934 16243 consensus_queue.cc:1035] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [LEADER]: Connected to new peer: Peer: permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:28.565986 15694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:51:28.566283 15694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:51:28.570106 15694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f: Sending DeleteTablet for 3 replicas of tablet d41c1e82dfce48c192e3be49b5a492c7
I20260806 04:51:28.571103 15834 tablet_service.cc:1514] Processing DeleteTablet for tablet d41c1e82dfce48c192e3be49b5a492c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:28 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:34076
I20260806 04:51:28.571498 15976 tablet_service.cc:1514] Processing DeleteTablet for tablet d41c1e82dfce48c192e3be49b5a492c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:28 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:38924
I20260806 04:51:28.571455 16117 tablet_service.cc:1514] Processing DeleteTablet for tablet d41c1e82dfce48c192e3be49b5a492c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:28 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:41976
I20260806 04:51:28.573257 15694 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:51:28.574025 15694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:51:28.584704 16253 tablet_replica.cc:330] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d: stopping tablet replica
I20260806 04:51:28.584971 15833 tablet_service.cc:1467] Processing CreateTablet for tablet cf2621170a564274918acba736773213 (DEFAULT_TABLE table=test-table [id=76426dda87744a8b8b36fc7a03c55ca4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:28.585139 16253 raft_consensus.cc:2229] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:51:28.585501 15833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2621170a564274918acba736773213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:28.585402 15975 tablet_service.cc:1467] Processing CreateTablet for tablet cf2621170a564274918acba736773213 (DEFAULT_TABLE table=test-table [id=76426dda87744a8b8b36fc7a03c55ca4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:28.585736 16253 raft_consensus.cc:2258] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:28.585845 15975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2621170a564274918acba736773213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:28.589124 16118 tablet_service.cc:1467] Processing CreateTablet for tablet cf2621170a564274918acba736773213 (DEFAULT_TABLE table=test-table [id=76426dda87744a8b8b36fc7a03c55ca4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:28.589488 16118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2621170a564274918acba736773213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:28.590547 16253 ts_tablet_manager.cc:1905] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:28.592047 16212 tablet_bootstrap.cc:492] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955: Bootstrap starting.
I20260806 04:51:28.593675 16212 tablet_bootstrap.cc:654] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:28.596532 16252 tablet_replica.cc:330] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98: stopping tablet replica
I20260806 04:51:28.596992 16252 raft_consensus.cc:2229] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:51:28.597385 16252 raft_consensus.cc:2258] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:28.601950 16211 tablet_bootstrap.cc:492] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98: Bootstrap starting.
I20260806 04:51:28.602674 16252 ts_tablet_manager.cc:1905] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:28.603997 16211 tablet_bootstrap.cc:654] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:28.604910 16213 tablet_bootstrap.cc:492] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d: Bootstrap starting.
I20260806 04:51:28.606155 16212 tablet_bootstrap.cc:492] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955: No bootstrap required, opened a new log
I20260806 04:51:28.606379 16212 ts_tablet_manager.cc:1397] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20260806 04:51:28.606395 16213 tablet_bootstrap.cc:654] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:28.607201 16212 raft_consensus.cc:348] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.607621 16212 raft_consensus.cc:374] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:28.607769 16212 raft_consensus.cc:729] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622d92e8a8a54d7990991907b515f955, State: Initialized, Role: FOLLOWER
I20260806 04:51:28.608160 16212 consensus_queue.cc:260] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.610054 16212 ts_tablet_manager.cc:1428] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260806 04:51:28.614699 16253 ts_tablet_manager.cc:1918] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:51:28.614858 16253 log.cc:1196] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/d41c1e82dfce48c192e3be49b5a492c7
I20260806 04:51:28.617255 16211 tablet_bootstrap.cc:492] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98: No bootstrap required, opened a new log
I20260806 04:51:28.617460 16211 ts_tablet_manager.cc:1397] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.005s
I20260806 04:51:28.618561 16211 raft_consensus.cc:348] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.618880 16211 raft_consensus.cc:374] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:28.618995 16211 raft_consensus.cc:729] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e1b681ae3b41d991e4feb81b768e98, State: Initialized, Role: FOLLOWER
I20260806 04:51:28.619341 16211 consensus_queue.cc:260] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.620129 16253 ts_tablet_manager.cc:1939] T d41c1e82dfce48c192e3be49b5a492c7 P 3f5f9dd993d04dc29ba74dfab215088d: Deleting consensus metadata
I20260806 04:51:28.621138 16251 tablet_replica.cc:330] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955: stopping tablet replica
I20260806 04:51:28.621644 16251 raft_consensus.cc:2229] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:51:28.622043 16251 raft_consensus.cc:2258] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:28.622076 15677 catalog_manager.cc:4841] TS 3f5f9dd993d04dc29ba74dfab215088d (127.13.148.131:39843): tablet d41c1e82dfce48c192e3be49b5a492c7 (table test-table [id=88bd669b94e74a809365b8d0b90fcc2e]) successfully deleted
I20260806 04:51:28.622366 16252 ts_tablet_manager.cc:1918] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:51:28.622572 16252 log.cc:1196] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/d41c1e82dfce48c192e3be49b5a492c7
I20260806 04:51:28.622700 16211 ts_tablet_manager.cc:1428] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260806 04:51:28.623059 16252 ts_tablet_manager.cc:1939] T d41c1e82dfce48c192e3be49b5a492c7 P c0e1b681ae3b41d991e4feb81b768e98: Deleting consensus metadata
I20260806 04:51:28.623687 16213 tablet_bootstrap.cc:492] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d: No bootstrap required, opened a new log
I20260806 04:51:28.623914 16213 ts_tablet_manager.cc:1397] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.001s
I20260806 04:51:28.624576 15679 catalog_manager.cc:4841] TS c0e1b681ae3b41d991e4feb81b768e98 (127.13.148.130:36263): tablet d41c1e82dfce48c192e3be49b5a492c7 (table test-table [id=88bd669b94e74a809365b8d0b90fcc2e]) successfully deleted
I20260806 04:51:28.624799 16213 raft_consensus.cc:348] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.625011 16213 raft_consensus.cc:374] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:28.625083 16213 raft_consensus.cc:729] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f5f9dd993d04dc29ba74dfab215088d, State: Initialized, Role: FOLLOWER
I20260806 04:51:28.625331 16213 consensus_queue.cc:260] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.625885 16213 ts_tablet_manager.cc:1428] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 04:51:28.626811 16251 ts_tablet_manager.cc:1905] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:28.633529 16218 raft_consensus.cc:482] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:51:28.633718 16251 ts_tablet_manager.cc:1918] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:51:28.633857 16251 log.cc:1196] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/d41c1e82dfce48c192e3be49b5a492c7
I20260806 04:51:28.633701 16218 raft_consensus.cc:504] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.634331 16251 ts_tablet_manager.cc:1939] T d41c1e82dfce48c192e3be49b5a492c7 P 622d92e8a8a54d7990991907b515f955: Deleting consensus metadata
I20260806 04:51:28.635210 16218 leader_election.cc:288] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 622d92e8a8a54d7990991907b515f955 (127.13.148.129:37893), 3f5f9dd993d04dc29ba74dfab215088d (127.13.148.131:39843)
I20260806 04:51:28.635805 15679 catalog_manager.cc:4841] TS 622d92e8a8a54d7990991907b515f955 (127.13.148.129:37893): tablet d41c1e82dfce48c192e3be49b5a492c7 (table test-table [id=88bd669b94e74a809365b8d0b90fcc2e]) successfully deleted
I20260806 04:51:28.645915 16205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.635589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:45019 (local address 127.13.148.129:37893)
0806 04:51:28.635750 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:28.635756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:28.637517 (+ 1761us) server_negotiation.cc:407] Connection header received
0806 04:51:28.637786 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:28.637791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:28.637865 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:28.637925 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:28.638704 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.640111 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.642409 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.643839 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.644031 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.644125 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:28.644473 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 04:51:28.644595 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:51:28.646286 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.635388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:37893 (local address 127.13.148.130:45019)
0806 04:51:28.637425 (+ 2037us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:28.637444 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:28.637635 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:28.638115 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:28.638123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:28.638149 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:28.638521 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.638533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.640251 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.640256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.642262 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.642276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.644060 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.644066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.644263 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.644299 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 04:51:28.645789 (+ 1490us) client_negotiation.cc:240] Negotiation successful
0806 04:51:28.645914 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1906,"thread_start_us":2592,"threads_started":1}
I20260806 04:51:28.646875 15854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2621170a564274918acba736773213" candidate_uuid: "c0e1b681ae3b41d991e4feb81b768e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622d92e8a8a54d7990991907b515f955" is_pre_election: true
I20260806 04:51:28.647099 15854 raft_consensus.cc:2454] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0e1b681ae3b41d991e4feb81b768e98 in term 0.
I20260806 04:51:28.647711 15927 leader_election.cc:302] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622d92e8a8a54d7990991907b515f955, c0e1b681ae3b41d991e4feb81b768e98; no voters:
I20260806 04:51:28.648087 16218 raft_consensus.cc:2781] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:51:28.648205 16218 raft_consensus.cc:482] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:51:28.648304 16218 raft_consensus.cc:3037] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:28.651118 16218 raft_consensus.cc:504] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.651793 16218 leader_election.cc:288] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [CANDIDATE]: Term 1 election: Requested vote from peers 622d92e8a8a54d7990991907b515f955 (127.13.148.129:37893), 3f5f9dd993d04dc29ba74dfab215088d (127.13.148.131:39843)
I20260806 04:51:28.652180 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.636173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:37019 (local address 127.13.148.131:39843)
0806 04:51:28.636315 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:28.636320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:28.644700 (+ 8380us) server_negotiation.cc:407] Connection header received
0806 04:51:28.644923 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:28.644926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:28.645041 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:28.645132 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:28.645774 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.646750 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.649788 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.650983 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.651112 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.651191 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:28.651743 (+ 552us) server_negotiation.cc:299] Negotiation successful
0806 04:51:28.651885 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:51:28.652275 15854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2621170a564274918acba736773213" candidate_uuid: "c0e1b681ae3b41d991e4feb81b768e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622d92e8a8a54d7990991907b515f955"
I20260806 04:51:28.652454 15854 raft_consensus.cc:3037] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:28.652819 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.636087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:39843 (local address 127.13.148.130:37019)
0806 04:51:28.644611 (+ 8524us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:28.644624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:28.644750 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:28.645263 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:28.645269 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:28.645289 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:28.645619 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.645628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.646898 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.646902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.649621 (+ 2719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.649639 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.651376 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.651382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.651536 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.651583 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 04:51:28.652371 (+ 788us) client_negotiation.cc:240] Negotiation successful
0806 04:51:28.652493 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8417,"thread_start_us":8242,"threads_started":1}
I20260806 04:51:28.653577 16143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2621170a564274918acba736773213" candidate_uuid: "c0e1b681ae3b41d991e4feb81b768e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f5f9dd993d04dc29ba74dfab215088d" is_pre_election: true
I20260806 04:51:28.653981 16143 raft_consensus.cc:2454] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0e1b681ae3b41d991e4feb81b768e98 in term 0.
I20260806 04:51:28.654449 16142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2621170a564274918acba736773213" candidate_uuid: "c0e1b681ae3b41d991e4feb81b768e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f5f9dd993d04dc29ba74dfab215088d"
I20260806 04:51:28.654654 16142 raft_consensus.cc:3037] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:28.654913 15854 raft_consensus.cc:2454] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0e1b681ae3b41d991e4feb81b768e98 in term 1.
I20260806 04:51:28.655283 15927 leader_election.cc:302] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622d92e8a8a54d7990991907b515f955, c0e1b681ae3b41d991e4feb81b768e98; no voters:
I20260806 04:51:28.655550 16218 raft_consensus.cc:2781] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:28.655933 16218 raft_consensus.cc:686] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 LEADER]: Becoming Leader. State: Replica: c0e1b681ae3b41d991e4feb81b768e98, State: Running, Role: LEADER
I20260806 04:51:28.656267 16218 consensus_queue.cc:237] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:28.657802 16142 raft_consensus.cc:2454] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0e1b681ae3b41d991e4feb81b768e98 in term 1.
I20260806 04:51:28.659489 15693 catalog_manager.cc:5495] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0e1b681ae3b41d991e4feb81b768e98 (127.13.148.130). New cstate: current_term: 1 leader_uuid: "c0e1b681ae3b41d991e4feb81b768e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } health_report { overall_health: UNKNOWN } } }
I20260806 04:51:28.692692 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.687218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.13.148.130:36263)
0806 04:51:28.687372 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:28.687379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:28.687511 (+ 132us) server_negotiation.cc:407] Connection header received
0806 04:51:28.687710 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:28.687713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:28.687779 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:28.687893 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:51:28.688268 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.689072 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.690722 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.691064 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.691205 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.691692 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:51:28.692125 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:51:28.692234 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:28.692354 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 04:51:28.692425 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:51:28.706776 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.701611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:35667 (local address 127.13.148.131:39843)
0806 04:51:28.701752 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:28.701756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:28.702146 (+ 390us) server_negotiation.cc:407] Connection header received
0806 04:51:28.702324 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:28.702327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:28.702400 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:28.702465 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:51:28.702986 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.703652 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.705393 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.705730 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.705813 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.706244 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:51:28.706324 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:51:28.706384 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:28.706510 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 04:51:28.706562 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 04:51:28.706974 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.701437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:39843 (local address 127.13.148.130:35667)
0806 04:51:28.702019 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:28.702045 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:28.702204 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:28.702514 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:28.702519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:28.702533 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:51:28.702856 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.702867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.703778 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.703783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.705275 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.705288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.705885 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.705892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.706045 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.706100 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:51:28.706412 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:51:28.706422 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 04:51:28.706488 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 04:51:28.706610 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":331,"threads_started":1}
I20260806 04:51:28.714259 15853 raft_consensus.cc:1261] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955 [term 1 FOLLOWER]: Refusing update from remote peer c0e1b681ae3b41d991e4feb81b768e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:28.714844 16259 consensus_queue.cc:1035] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:28.715636 16142 raft_consensus.cc:1261] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d [term 1 FOLLOWER]: Refusing update from remote peer c0e1b681ae3b41d991e4feb81b768e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:28.716544 16259 consensus_queue.cc:1035] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:28.742367 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.736610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:33021 (local address 127.13.148.130:36263)
0806 04:51:28.737150 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:28.737155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:28.737167 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:51:28.737209 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:28.737212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:28.737262 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:28.737359 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:51:28.737764 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.738863 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.740547 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.741219 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.741360 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.741748 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:51:28.741861 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:51:28.741950 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:28.742041 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 04:51:28.742118 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 04:51:28.742408 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:28.736629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:36263 (local address 127.13.148.131:33021)
0806 04:51:28.736878 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:28.736895 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:28.736989 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:28.737378 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:28.737382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:28.737394 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:51:28.737601 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.737618 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.738999 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.739002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.740367 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:28.740403 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:28.741395 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:28.741400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:28.741543 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:28.741575 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:51:28.741957 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:51:28.741965 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 04:51:28.742017 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 04:51:28.742118 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 04:51:28.743626 16238 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 04:51:30.336978 15694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:51:30.337198 15694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:51:30.340934 15694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f: Sending DeleteTablet for 3 replicas of tablet cf2621170a564274918acba736773213
I20260806 04:51:30.341653 15834 tablet_service.cc:1514] Processing DeleteTablet for tablet cf2621170a564274918acba736773213 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:30 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:34076
I20260806 04:51:30.341696 15975 tablet_service.cc:1514] Processing DeleteTablet for tablet cf2621170a564274918acba736773213 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:30 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:38924
I20260806 04:51:30.341813 16118 tablet_service.cc:1514] Processing DeleteTablet for tablet cf2621170a564274918acba736773213 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:30 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:41976
I20260806 04:51:30.357282 16292 tablet_replica.cc:330] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955: stopping tablet replica
I20260806 04:51:30.357636 16292 raft_consensus.cc:2229] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:51:30.358011 16292 raft_consensus.cc:2258] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:30.358793 16293 tablet_replica.cc:330] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98: stopping tablet replica
I20260806 04:51:30.358973 16294 tablet_replica.cc:330] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d: stopping tablet replica
I20260806 04:51:30.359176 16293 raft_consensus.cc:2229] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:51:30.359223 16294 raft_consensus.cc:2229] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:51:30.359447 16294 raft_consensus.cc:2258] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:30.359498 16292 ts_tablet_manager.cc:1905] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:30.359872 16293 raft_consensus.cc:2258] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:30.360442 16294 ts_tablet_manager.cc:1905] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:30.360896 16293 ts_tablet_manager.cc:1905] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:30.365836 16292 ts_tablet_manager.cc:1918] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 04:51:30.365969 16292 log.cc:1196] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/cf2621170a564274918acba736773213
I20260806 04:51:30.366408 16292 ts_tablet_manager.cc:1939] T cf2621170a564274918acba736773213 P 622d92e8a8a54d7990991907b515f955: Deleting consensus metadata
I20260806 04:51:30.367326 16293 ts_tablet_manager.cc:1918] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 04:51:30.367544 16293 log.cc:1196] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/cf2621170a564274918acba736773213
I20260806 04:51:30.367529 15679 catalog_manager.cc:4841] TS 622d92e8a8a54d7990991907b515f955 (127.13.148.129:37893): tablet cf2621170a564274918acba736773213 (table test-table [id=76426dda87744a8b8b36fc7a03c55ca4]) successfully deleted
I20260806 04:51:30.367683 16294 ts_tablet_manager.cc:1918] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 04:51:30.367820 16294 log.cc:1196] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/cf2621170a564274918acba736773213
I20260806 04:51:30.368188 16293 ts_tablet_manager.cc:1939] T cf2621170a564274918acba736773213 P c0e1b681ae3b41d991e4feb81b768e98: Deleting consensus metadata
I20260806 04:51:30.368290 16294 ts_tablet_manager.cc:1939] T cf2621170a564274918acba736773213 P 3f5f9dd993d04dc29ba74dfab215088d: Deleting consensus metadata
I20260806 04:51:30.369295 15677 catalog_manager.cc:4841] TS 3f5f9dd993d04dc29ba74dfab215088d (127.13.148.131:39843): tablet cf2621170a564274918acba736773213 (table test-table [id=76426dda87744a8b8b36fc7a03c55ca4]) successfully deleted
I20260806 04:51:30.369338 15679 catalog_manager.cc:4841] TS c0e1b681ae3b41d991e4feb81b768e98 (127.13.148.130:36263): tablet cf2621170a564274918acba736773213 (table test-table [id=76426dda87744a8b8b36fc7a03c55ca4]) successfully deleted
Aug 06 04:51:30 dist-test-slave-1x4g krb5kdc[15648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991890, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 04:51:30 dist-test-slave-1x4g krb5kdc[15648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991890, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:30.402100 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:30.383875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39750 (local address 127.13.148.190:41281)
0806 04:51:30.384441 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:30.384459 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:30.384535 (+ 76us) server_negotiation.cc:407] Connection header received
0806 04:51:30.384731 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:30.384734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:30.384854 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:30.384956 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:30.386200 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.386913 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.388110 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.388547 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.388688 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:30.396837 (+ 8149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:30.396858 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:30.396861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:30.396896 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:30.399623 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:30.400194 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:30.400197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:30.400199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:30.400344 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:30.400680 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:30.400683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:30.400684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:30.401056 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:30.401250 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:30.401485 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 04:51:30.401654 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":277,"threads_started":1}
I20260806 04:51:30.405109 15694 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:51:30.405689 15694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:51:30.411903 15834 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce50f30a0424f52ab4c347ef35849f4 (DEFAULT_TABLE table=test-table [id=f8f8d948b22f440389bf7c0ea571e558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:30.411963 15975 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce50f30a0424f52ab4c347ef35849f4 (DEFAULT_TABLE table=test-table [id=f8f8d948b22f440389bf7c0ea571e558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:30.412340 15834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce50f30a0424f52ab4c347ef35849f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:30.412357 15975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce50f30a0424f52ab4c347ef35849f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:30.412312 16118 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce50f30a0424f52ab4c347ef35849f4 (DEFAULT_TABLE table=test-table [id=f8f8d948b22f440389bf7c0ea571e558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:30.412907 16118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce50f30a0424f52ab4c347ef35849f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:30.419533 16303 tablet_bootstrap.cc:492] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d: Bootstrap starting.
I20260806 04:51:30.421159 16303 tablet_bootstrap.cc:654] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:30.421330 16305 tablet_bootstrap.cc:492] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955: Bootstrap starting.
I20260806 04:51:30.423256 16305 tablet_bootstrap.cc:654] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:30.424069 16303 tablet_bootstrap.cc:492] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d: No bootstrap required, opened a new log
I20260806 04:51:30.424242 16303 ts_tablet_manager.cc:1397] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 04:51:30.424844 16304 tablet_bootstrap.cc:492] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98: Bootstrap starting.
I20260806 04:51:30.425352 16303 raft_consensus.cc:348] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.425689 16303 raft_consensus.cc:374] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:30.425796 16303 raft_consensus.cc:729] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f5f9dd993d04dc29ba74dfab215088d, State: Initialized, Role: FOLLOWER
I20260806 04:51:30.426306 16304 tablet_bootstrap.cc:654] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:30.426146 16303 consensus_queue.cc:260] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.426963 16303 ts_tablet_manager.cc:1428] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260806 04:51:30.428529 16305 tablet_bootstrap.cc:492] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955: No bootstrap required, opened a new log
I20260806 04:51:30.428716 16305 ts_tablet_manager.cc:1397] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20260806 04:51:30.429659 16305 raft_consensus.cc:348] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.429877 16304 tablet_bootstrap.cc:492] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98: No bootstrap required, opened a new log
I20260806 04:51:30.429899 16305 raft_consensus.cc:374] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:30.430024 16304 ts_tablet_manager.cc:1397] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260806 04:51:30.430025 16305 raft_consensus.cc:729] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622d92e8a8a54d7990991907b515f955, State: Initialized, Role: FOLLOWER
I20260806 04:51:30.430315 16305 consensus_queue.cc:260] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.430972 16304 raft_consensus.cc:348] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.431200 16304 raft_consensus.cc:374] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:30.431226 16305 ts_tablet_manager.cc:1428] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 04:51:30.431298 16304 raft_consensus.cc:729] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e1b681ae3b41d991e4feb81b768e98, State: Initialized, Role: FOLLOWER
I20260806 04:51:30.431543 16304 consensus_queue.cc:260] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.432111 16304 ts_tablet_manager.cc:1428] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 04:51:30.437559 16309 raft_consensus.cc:482] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:51:30.437736 16309 raft_consensus.cc:504] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.439097 16309 leader_election.cc:288] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0e1b681ae3b41d991e4feb81b768e98 (127.13.148.130:36263), 3f5f9dd993d04dc29ba74dfab215088d (127.13.148.131:39843)
I20260806 04:51:30.457000 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:30.439442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:44637 (local address 127.13.148.131:39843)
0806 04:51:30.451800 (+ 12358us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:30.451804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:30.451819 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:51:30.451866 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:30.451869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:30.451924 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:30.452036 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:30.452756 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.453699 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.455244 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.455972 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.456149 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:30.456213 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:30.456506 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 04:51:30.456651 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12247,"thread_start_us":12147,"threads_started":1}
I20260806 04:51:30.457036 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:30.439333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:39843 (local address 127.13.148.129:44637)
0806 04:51:30.451188 (+ 11855us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:30.451211 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:30.451341 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:30.452307 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:30.452311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:30.452338 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:30.452552 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:30.452560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.453837 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.453841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:30.455103 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:30.455114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.456155 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.456160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:30.456345 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:30.456370 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 04:51:30.456474 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 04:51:30.456604 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11704,"thread_start_us":12509,"threads_started":1}
I20260806 04:51:30.457198 16310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:30.439372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:36263 (local address 127.13.148.129:43723)
0806 04:51:30.452160 (+ 12788us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:30.452176 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:30.452290 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:30.452664 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:30.452669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:30.452694 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:30.453115 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:30.453126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.454073 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.454082 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:30.455276 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:30.455287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.456384 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.456390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:30.456534 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:30.456555 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 04:51:30.456678 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 04:51:30.456823 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12619,"thread_start_us":11464,"threads_started":1}
I20260806 04:51:30.457195 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:30.439363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:43723 (local address 127.13.148.130:36263)
0806 04:51:30.451448 (+ 12085us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:30.451454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:30.452402 (+ 948us) server_negotiation.cc:407] Connection header received
0806 04:51:30.452472 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:30.452476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:30.452545 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:30.452633 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:30.453268 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.453924 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.455413 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.456154 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.456331 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:30.456404 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:30.456713 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 04:51:30.456861 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11930,"thread_start_us":12792,"threads_started":1}
I20260806 04:51:30.457847 15996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce50f30a0424f52ab4c347ef35849f4" candidate_uuid: "622d92e8a8a54d7990991907b515f955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e1b681ae3b41d991e4feb81b768e98" is_pre_election: true
I20260806 04:51:30.458120 15996 raft_consensus.cc:2454] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 622d92e8a8a54d7990991907b515f955 in term 0.
I20260806 04:51:30.458252 16143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce50f30a0424f52ab4c347ef35849f4" candidate_uuid: "622d92e8a8a54d7990991907b515f955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f5f9dd993d04dc29ba74dfab215088d" is_pre_election: true
I20260806 04:51:30.458500 16143 raft_consensus.cc:2454] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 622d92e8a8a54d7990991907b515f955 in term 0.
I20260806 04:51:30.458680 15784 leader_election.cc:302] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622d92e8a8a54d7990991907b515f955, c0e1b681ae3b41d991e4feb81b768e98; no voters:
I20260806 04:51:30.459125 16309 raft_consensus.cc:2781] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:51:30.459281 16309 raft_consensus.cc:482] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:51:30.459414 16309 raft_consensus.cc:3037] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:30.461465 16309 raft_consensus.cc:504] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.462086 16309 leader_election.cc:288] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [CANDIDATE]: Term 1 election: Requested vote from peers c0e1b681ae3b41d991e4feb81b768e98 (127.13.148.130:36263), 3f5f9dd993d04dc29ba74dfab215088d (127.13.148.131:39843)
I20260806 04:51:30.462493 15996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce50f30a0424f52ab4c347ef35849f4" candidate_uuid: "622d92e8a8a54d7990991907b515f955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e1b681ae3b41d991e4feb81b768e98"
I20260806 04:51:30.462579 16143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce50f30a0424f52ab4c347ef35849f4" candidate_uuid: "622d92e8a8a54d7990991907b515f955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f5f9dd993d04dc29ba74dfab215088d"
I20260806 04:51:30.462716 15996 raft_consensus.cc:3037] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:30.462826 16143 raft_consensus.cc:3037] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:30.464707 15996 raft_consensus.cc:2454] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 622d92e8a8a54d7990991907b515f955 in term 1.
I20260806 04:51:30.464980 16143 raft_consensus.cc:2454] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 622d92e8a8a54d7990991907b515f955 in term 1.
I20260806 04:51:30.465147 15784 leader_election.cc:302] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622d92e8a8a54d7990991907b515f955, c0e1b681ae3b41d991e4feb81b768e98; no voters:
I20260806 04:51:30.465418 16309 raft_consensus.cc:2781] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:30.465878 16309 raft_consensus.cc:686] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [term 1 LEADER]: Becoming Leader. State: Replica: 622d92e8a8a54d7990991907b515f955, State: Running, Role: LEADER
I20260806 04:51:30.466259 16309 consensus_queue.cc:237] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.469444 15694 catalog_manager.cc:5495] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 reported cstate change: term changed from 0 to 1, leader changed from <none> to 622d92e8a8a54d7990991907b515f955 (127.13.148.129). New cstate: current_term: 1 leader_uuid: "622d92e8a8a54d7990991907b515f955" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } health_report { overall_health: UNKNOWN } } }
I20260806 04:51:30.492583 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:30.486741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.13.148.129:37893)
0806 04:51:30.487294 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:30.487299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:30.487324 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:51:30.487394 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:30.487397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:30.487463 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:30.487616 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:51:30.488063 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.488978 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.490400 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.490810 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.490939 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:30.491424 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:51:30.491973 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:51:30.492122 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:30.492176 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 04:51:30.492248 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":288,"threads_started":1}
I20260806 04:51:30.505787 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:30.501759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:39547 (local address 127.13.148.131:39843)
0806 04:51:30.501920 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:30.501924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:30.502100 (+ 176us) server_negotiation.cc:407] Connection header received
0806 04:51:30.502229 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:30.502232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:30.502290 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:30.502357 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:51:30.502756 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.503367 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.504489 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.504866 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.504949 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:30.505311 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:51:30.505393 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:51:30.505449 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:30.505565 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 04:51:30.505619 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:51:30.505858 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:30.501488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:39843 (local address 127.13.148.129:39547)
0806 04:51:30.502018 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:30.502036 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:30.502135 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:30.502382 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:30.502385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:30.502393 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:51:30.502638 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:30.502644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.503475 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.503478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:30.504388 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:30.504397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.504995 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.504999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:30.505136 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:30.505185 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:51:30.505473 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:51:30.505480 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 04:51:30.505533 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 04:51:30.505615 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":270,"threads_started":1}
I20260806 04:51:30.513430 16143 raft_consensus.cc:1261] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d [term 1 FOLLOWER]: Refusing update from remote peer 622d92e8a8a54d7990991907b515f955: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:30.513510 15996 raft_consensus.cc:1261] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 FOLLOWER]: Refusing update from remote peer 622d92e8a8a54d7990991907b515f955: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:30.514062 16309 consensus_queue.cc:1035] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:30.514317 16314 consensus_queue.cc:1035] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:30.538421 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:30.533187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:43249 (local address 127.13.148.129:37893)
0806 04:51:30.533339 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:30.533344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:30.533887 (+ 543us) server_negotiation.cc:407] Connection header received
0806 04:51:30.534087 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:30.534092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:30.534162 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:30.534249 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:51:30.534696 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.535337 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.536987 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.537477 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.537569 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:30.537933 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:51:30.538022 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:51:30.538086 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:30.538186 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 04:51:30.538244 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:51:30.538589 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:30.533109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:37893 (local address 127.13.148.131:43249)
0806 04:51:30.533784 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:30.533801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:30.533958 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:30.534268 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:30.534273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:30.534285 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:51:30.534548 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:30.534557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.535491 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.535497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:30.536833 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:30.536847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.537622 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.537627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:30.537761 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:30.537795 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:51:30.538107 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:51:30.538115 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 04:51:30.538172 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 04:51:30.538262 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":406,"threads_started":1}
I20260806 04:51:30.568482 15693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:51:30.568747 15693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:51:30.571857 15693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f: Sending DeleteTablet for 3 replicas of tablet 7ce50f30a0424f52ab4c347ef35849f4
I20260806 04:51:30.572649 15834 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ce50f30a0424f52ab4c347ef35849f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:30 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:34076
I20260806 04:51:30.572746 16118 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ce50f30a0424f52ab4c347ef35849f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:30 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:41976
I20260806 04:51:30.572940 16292 tablet_replica.cc:330] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955: stopping tablet replica
I20260806 04:51:30.573012 16294 tablet_replica.cc:330] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d: stopping tablet replica
I20260806 04:51:30.573141 16292 raft_consensus.cc:2229] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:51:30.573179 16294 raft_consensus.cc:2229] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:51:30.573339 16294 raft_consensus.cc:2258] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:30.573617 16292 raft_consensus.cc:2258] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:30.574311 16294 ts_tablet_manager.cc:1905] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:30.574502 15975 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ce50f30a0424f52ab4c347ef35849f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:30 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:38924
I20260806 04:51:30.575496 15693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:51:30.576036 15693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:51:30.580651 16294 ts_tablet_manager.cc:1918] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:51:30.581009 16293 tablet_replica.cc:330] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98: stopping tablet replica
I20260806 04:51:30.581053 16294 log.cc:1196] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/7ce50f30a0424f52ab4c347ef35849f4
I20260806 04:51:30.581224 16293 raft_consensus.cc:2229] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:51:30.581410 16293 raft_consensus.cc:2258] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:30.581488 16294 ts_tablet_manager.cc:1939] T 7ce50f30a0424f52ab4c347ef35849f4 P 3f5f9dd993d04dc29ba74dfab215088d: Deleting consensus metadata
I20260806 04:51:30.582675 16292 ts_tablet_manager.cc:1905] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:30.585587 15834 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6586f7afcc4f9393bdaf3fdf3bd488 (DEFAULT_TABLE table=test-table [id=47995c9e0a854b37a6c566b6eb3a01a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:30.586066 15834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6586f7afcc4f9393bdaf3fdf3bd488. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:30.586189 16118 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6586f7afcc4f9393bdaf3fdf3bd488 (DEFAULT_TABLE table=test-table [id=47995c9e0a854b37a6c566b6eb3a01a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:30.586509 16293 ts_tablet_manager.cc:1905] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:30.586536 16118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6586f7afcc4f9393bdaf3fdf3bd488. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:30.589401 15975 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6586f7afcc4f9393bdaf3fdf3bd488 (DEFAULT_TABLE table=test-table [id=47995c9e0a854b37a6c566b6eb3a01a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:30.589910 15975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6586f7afcc4f9393bdaf3fdf3bd488. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:30.593426 16293 ts_tablet_manager.cc:1918] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:51:30.593575 16293 log.cc:1196] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/7ce50f30a0424f52ab4c347ef35849f4
I20260806 04:51:30.593981 16293 ts_tablet_manager.cc:1939] T 7ce50f30a0424f52ab4c347ef35849f4 P c0e1b681ae3b41d991e4feb81b768e98: Deleting consensus metadata
I20260806 04:51:30.595137 15677 catalog_manager.cc:4841] TS 3f5f9dd993d04dc29ba74dfab215088d (127.13.148.131:39843): tablet 7ce50f30a0424f52ab4c347ef35849f4 (table test-table [id=f8f8d948b22f440389bf7c0ea571e558]) successfully deleted
I20260806 04:51:30.597379 16292 ts_tablet_manager.cc:1918] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:51:30.597563 16292 log.cc:1196] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/7ce50f30a0424f52ab4c347ef35849f4
I20260806 04:51:30.597640 15679 catalog_manager.cc:4841] TS c0e1b681ae3b41d991e4feb81b768e98 (127.13.148.130:36263): tablet 7ce50f30a0424f52ab4c347ef35849f4 (table test-table [id=f8f8d948b22f440389bf7c0ea571e558]) successfully deleted
I20260806 04:51:30.598058 16292 ts_tablet_manager.cc:1939] T 7ce50f30a0424f52ab4c347ef35849f4 P 622d92e8a8a54d7990991907b515f955: Deleting consensus metadata
I20260806 04:51:30.599109 15679 catalog_manager.cc:4841] TS 622d92e8a8a54d7990991907b515f955 (127.13.148.129:37893): tablet 7ce50f30a0424f52ab4c347ef35849f4 (table test-table [id=f8f8d948b22f440389bf7c0ea571e558]) successfully deleted
I20260806 04:51:30.600111 16303 tablet_bootstrap.cc:492] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d: Bootstrap starting.
I20260806 04:51:30.600492 16305 tablet_bootstrap.cc:492] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955: Bootstrap starting.
I20260806 04:51:30.601619 16303 tablet_bootstrap.cc:654] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:30.601936 16305 tablet_bootstrap.cc:654] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:30.605348 16304 tablet_bootstrap.cc:492] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98: Bootstrap starting.
I20260806 04:51:30.606884 16304 tablet_bootstrap.cc:654] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:30.609994 16303 tablet_bootstrap.cc:492] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d: No bootstrap required, opened a new log
I20260806 04:51:30.610117 16305 tablet_bootstrap.cc:492] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955: No bootstrap required, opened a new log
I20260806 04:51:30.610172 16303 ts_tablet_manager.cc:1397] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260806 04:51:30.610255 16305 ts_tablet_manager.cc:1397] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20260806 04:51:30.610862 16305 raft_consensus.cc:348] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.610893 16303 raft_consensus.cc:348] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.611078 16305 raft_consensus.cc:374] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:30.611129 16305 raft_consensus.cc:729] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622d92e8a8a54d7990991907b515f955, State: Initialized, Role: FOLLOWER
I20260806 04:51:30.611115 16303 raft_consensus.cc:374] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:30.611198 16303 raft_consensus.cc:729] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f5f9dd993d04dc29ba74dfab215088d, State: Initialized, Role: FOLLOWER
I20260806 04:51:30.611415 16303 consensus_queue.cc:260] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.611388 16305 consensus_queue.cc:260] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.611845 16305 ts_tablet_manager.cc:1428] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260806 04:51:30.611847 16303 ts_tablet_manager.cc:1428] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260806 04:51:30.618315 16304 tablet_bootstrap.cc:492] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98: No bootstrap required, opened a new log
I20260806 04:51:30.618487 16304 ts_tablet_manager.cc:1397] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260806 04:51:30.619216 16304 raft_consensus.cc:348] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.619437 16304 raft_consensus.cc:374] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:30.619519 16304 raft_consensus.cc:729] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e1b681ae3b41d991e4feb81b768e98, State: Initialized, Role: FOLLOWER
I20260806 04:51:30.619742 16304 consensus_queue.cc:260] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.620285 16304 ts_tablet_manager.cc:1428] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 04:51:30.695030 16282 raft_consensus.cc:482] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:51:30.695273 16282 raft_consensus.cc:504] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.696022 16282 leader_election.cc:288] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 622d92e8a8a54d7990991907b515f955 (127.13.148.129:37893), 3f5f9dd993d04dc29ba74dfab215088d (127.13.148.131:39843)
I20260806 04:51:30.696476 15854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6586f7afcc4f9393bdaf3fdf3bd488" candidate_uuid: "c0e1b681ae3b41d991e4feb81b768e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622d92e8a8a54d7990991907b515f955" is_pre_election: true
I20260806 04:51:30.696580 16143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6586f7afcc4f9393bdaf3fdf3bd488" candidate_uuid: "c0e1b681ae3b41d991e4feb81b768e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f5f9dd993d04dc29ba74dfab215088d" is_pre_election: true
I20260806 04:51:30.696815 15854 raft_consensus.cc:2454] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0e1b681ae3b41d991e4feb81b768e98 in term 0.
I20260806 04:51:30.696864 16143 raft_consensus.cc:2454] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0e1b681ae3b41d991e4feb81b768e98 in term 0.
I20260806 04:51:30.697269 15925 leader_election.cc:302] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f5f9dd993d04dc29ba74dfab215088d, c0e1b681ae3b41d991e4feb81b768e98; no voters:
I20260806 04:51:30.697647 16282 raft_consensus.cc:2781] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:51:30.697820 16282 raft_consensus.cc:482] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:51:30.697932 16282 raft_consensus.cc:3037] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:30.700876 16282 raft_consensus.cc:504] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.701514 16282 leader_election.cc:288] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [CANDIDATE]: Term 1 election: Requested vote from peers 622d92e8a8a54d7990991907b515f955 (127.13.148.129:37893), 3f5f9dd993d04dc29ba74dfab215088d (127.13.148.131:39843)
I20260806 04:51:30.701946 15854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6586f7afcc4f9393bdaf3fdf3bd488" candidate_uuid: "c0e1b681ae3b41d991e4feb81b768e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622d92e8a8a54d7990991907b515f955"
I20260806 04:51:30.702170 15854 raft_consensus.cc:3037] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:30.702247 16143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6586f7afcc4f9393bdaf3fdf3bd488" candidate_uuid: "c0e1b681ae3b41d991e4feb81b768e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f5f9dd993d04dc29ba74dfab215088d"
I20260806 04:51:30.702477 16143 raft_consensus.cc:3037] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:30.704244 16143 raft_consensus.cc:2454] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0e1b681ae3b41d991e4feb81b768e98 in term 1.
I20260806 04:51:30.704607 15925 leader_election.cc:302] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f5f9dd993d04dc29ba74dfab215088d, c0e1b681ae3b41d991e4feb81b768e98; no voters:
I20260806 04:51:30.704882 16282 raft_consensus.cc:2781] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:30.705040 15854 raft_consensus.cc:2454] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0e1b681ae3b41d991e4feb81b768e98 in term 1.
I20260806 04:51:30.705044 16282 raft_consensus.cc:686] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 LEADER]: Becoming Leader. State: Replica: c0e1b681ae3b41d991e4feb81b768e98, State: Running, Role: LEADER
I20260806 04:51:30.705593 16282 consensus_queue.cc:237] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } }
I20260806 04:51:30.708330 15693 catalog_manager.cc:5495] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0e1b681ae3b41d991e4feb81b768e98 (127.13.148.130). New cstate: current_term: 1 leader_uuid: "c0e1b681ae3b41d991e4feb81b768e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0e1b681ae3b41d991e4feb81b768e98" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 } health_report { overall_health: UNKNOWN } } }
I20260806 04:51:30.734814 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:30.730443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.13.148.130:36263)
0806 04:51:30.730590 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:30.730594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:30.730655 (+ 61us) server_negotiation.cc:407] Connection header received
0806 04:51:30.730839 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:30.730842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:30.730894 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:30.730962 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:51:30.731336 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.731964 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.733298 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:30.733674 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:30.733802 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:30.734169 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:51:30.734257 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:51:30.734349 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:30.734481 (+ 132us) server_negotiation.cc:299] Negotiation successful
0806 04:51:30.734566 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:51:30.743188 16143 raft_consensus.cc:1261] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d [term 1 FOLLOWER]: Refusing update from remote peer c0e1b681ae3b41d991e4feb81b768e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:30.743369 15854 raft_consensus.cc:1261] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955 [term 1 FOLLOWER]: Refusing update from remote peer c0e1b681ae3b41d991e4feb81b768e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:30.743649 16282 consensus_queue.cc:1035] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f5f9dd993d04dc29ba74dfab215088d" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 39843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:30.743970 16287 consensus_queue.cc:1035] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "622d92e8a8a54d7990991907b515f955" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 04:51:30.767536 16273 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 04:51:32.351352 15693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:51:32.351610 15693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:51:32.355093 15693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 295d1ba13dc34e5fb0be41631f1ec89f: Sending DeleteTablet for 3 replicas of tablet 2d6586f7afcc4f9393bdaf3fdf3bd488
I20260806 04:51:32.355768 15834 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d6586f7afcc4f9393bdaf3fdf3bd488 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:32 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:34076
I20260806 04:51:32.355820 16118 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d6586f7afcc4f9393bdaf3fdf3bd488 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:32 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:41976
I20260806 04:51:32.356142 15975 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d6586f7afcc4f9393bdaf3fdf3bd488 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:32 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:38924
I20260806 04:51:32.356366 16357 tablet_replica.cc:330] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955: stopping tablet replica
I20260806 04:51:32.356350 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 15766
I20260806 04:51:32.356706 16357 raft_consensus.cc:2229] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:51:32.357110 16357 raft_consensus.cc:2258] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:32.358918 16359 tablet_replica.cc:330] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98: stopping tablet replica
I20260806 04:51:32.359141 16359 raft_consensus.cc:2229] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:51:32.359244 16357 ts_tablet_manager.cc:1905] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:32.359503 16359 raft_consensus.cc:2258] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:32.360366 16359 ts_tablet_manager.cc:1905] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:32.365260 16357 ts_tablet_manager.cc:1918] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 04:51:32.365402 16357 log.cc:1196] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/2d6586f7afcc4f9393bdaf3fdf3bd488
I20260806 04:51:32.365831 16357 ts_tablet_manager.cc:1939] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 622d92e8a8a54d7990991907b515f955: Deleting consensus metadata
I20260806 04:51:32.366796 15679 catalog_manager.cc:4841] TS 622d92e8a8a54d7990991907b515f955 (127.13.148.129:37893): tablet 2d6586f7afcc4f9393bdaf3fdf3bd488 (table test-table [id=47995c9e0a854b37a6c566b6eb3a01a8]) successfully deleted
I20260806 04:51:32.367982 16358 tablet_replica.cc:330] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d: stopping tablet replica
I20260806 04:51:32.368180 16358 raft_consensus.cc:2229] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:51:32.368389 16359 ts_tablet_manager.cc:1918] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 04:51:32.368415 16358 raft_consensus.cc:2258] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:32.368530 16359 log.cc:1196] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/2d6586f7afcc4f9393bdaf3fdf3bd488
I20260806 04:51:32.369059 16359 ts_tablet_manager.cc:1939] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P c0e1b681ae3b41d991e4feb81b768e98: Deleting consensus metadata
I20260806 04:51:32.369361 16358 ts_tablet_manager.cc:1905] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:32.369900 15679 catalog_manager.cc:4841] TS c0e1b681ae3b41d991e4feb81b768e98 (127.13.148.130:36263): tablet 2d6586f7afcc4f9393bdaf3fdf3bd488 (table test-table [id=47995c9e0a854b37a6c566b6eb3a01a8]) successfully deleted
Aug 06 04:51:32 dist-test-slave-1x4g krb5kdc[15648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991890, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:51:32.375085 16358 ts_tablet_manager.cc:1918] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 04:51:32.375218 16358 log.cc:1196] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/2d6586f7afcc4f9393bdaf3fdf3bd488
I20260806 04:51:32.375639 16358 ts_tablet_manager.cc:1939] T 2d6586f7afcc4f9393bdaf3fdf3bd488 P 3f5f9dd993d04dc29ba74dfab215088d: Deleting consensus metadata
I20260806 04:51:32.376521 15677 catalog_manager.cc:4841] TS 3f5f9dd993d04dc29ba74dfab215088d (127.13.148.131:39843): tablet 2d6586f7afcc4f9393bdaf3fdf3bd488 (table test-table [id=47995c9e0a854b37a6c566b6eb3a01a8]) successfully deleted
I20260806 04:51:32.377511 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:32.357660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42802 (local address 127.13.148.129:37893)
0806 04:51:32.361081 (+ 3421us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:32.361087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:32.361105 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:51:32.361149 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:32.361154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:32.361235 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:32.361339 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:32.363124 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:32.364185 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:32.366040 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:32.366573 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:32.366676 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:32.371582 (+ 4906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:32.371596 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:32.371615 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:32.371652 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:32.375037 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:32.375686 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:32.375696 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:32.375698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:32.375848 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:32.376151 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:32.376155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:32.376158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:32.376538 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:32.376903 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:32.377111 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 04:51:32.377260 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3295,"thread_start_us":3218,"threads_started":1}
W20260806 04:51:32.378158 15899 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42802
W20260806 04:51:32.378681 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 15766: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:51:32.378877 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 15766
I20260806 04:51:32.402004 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 15909
Aug 06 04:51:32 dist-test-slave-1x4g krb5kdc[15648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991890, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:51:32.415329 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:32.402613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.13.148.130:36263)
0806 04:51:32.403070 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:32.403074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:32.403089 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:51:32.403132 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:32.403135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:32.403185 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:32.403260 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:32.404302 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:32.405100 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:32.406193 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:32.406556 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:32.406642 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:32.410418 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:32.410428 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:32.410440 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:32.410475 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:32.413084 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:32.413678 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:32.413685 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:32.413687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:32.413874 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:32.414164 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:32.414167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:32.414169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:32.414481 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:32.414671 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:32.414969 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 04:51:32.415112 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":255,"threads_started":1}
W20260806 04:51:32.415917 16041 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57162
W20260806 04:51:32.416496 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 15909: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:51:32.416671 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 15909
I20260806 04:51:32.439697 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 16050
Aug 06 04:51:32 dist-test-slave-1x4g krb5kdc[15648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991890, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
I20260806 04:51:32.453109 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:32.440291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.13.148.131:39843)
0806 04:51:32.440742 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:32.440746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:32.440759 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:51:32.440846 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:32.440849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:32.440912 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:32.441018 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:32.442136 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:32.442800 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:32.443911 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:32.444287 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:32.444416 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:32.448230 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:32.448243 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:32.448259 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:32.448289 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:32.450800 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:32.451415 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:32.451422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:32.451424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:32.451537 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:32.451844 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:32.451847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:32.451849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:32.452161 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:32.452345 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:32.452661 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 04:51:32.452878 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":221,"threads_started":1}
W20260806 04:51:32.453684 16184 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43750
W20260806 04:51:32.454218 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 16050: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:51:32.454341 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 16050
I20260806 04:51:32.479147 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 15661
I20260806 04:51:32.480086 15735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:32.604910 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 15661
2026-08-06T04:51:32Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7598 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:51:32 dist-test-slave-1x4g krb5kdc[16367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:51:32 dist-test-slave-1x4g krb5kdc[16367](info): set up 2 sockets
Aug 06 04:51:32 dist-test-slave-1x4g krb5kdc[16367](info): commencing operation
krb5kdc: starting...
W20260806 04:51:33.618853 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:51:33 dist-test-slave-1x4g krb5kdc[16367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:51:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:51:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:51:33.792429 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:35439
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:42657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:35439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:34.086232 16381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:34.086527 16381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:34.086673 16381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:34.092976 16381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:51:34.093092 16381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:34.093186 16381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:34.093268 16381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:51:34.093350 16381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:51:34.101928 16381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:42657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:35439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:35439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:34.103120 16381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:34.104399 16381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:34.115567 16389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:34.115615 16386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:34.115643 16387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:34.116277 16381 server_base.cc:1034] running on GCE node
I20260806 04:51:34.117009 16381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:34.118325 16381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:34.119519 16381 hybrid_clock.cc:648] HybridClock initialized: now 1785991894119476 us; error 37 us; skew 500 ppm
I20260806 04:51:34.119786 16381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:34 dist-test-slave-1x4g krb5kdc[16367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991894, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:34.122735 16381 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:51:34.124063 16381 webserver.cc:460] Webserver started at http://127.13.148.190:40417/ using document root <none> and password file <none>
I20260806 04:51:34.124646 16381 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:34.124735 16381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:34.125036 16381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:34.127593 16381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "5eeb73e54fad4f20b7aa7b82b5e970db"
format_stamp: "Formatted at 2026-08-06 04:51:34 on dist-test-slave-1x4g"
server_key: "a1dff734c998098b693eb8772b336ee4"
server_key_iv: "7e8da313987fd596cf2a682993a95f49"
server_key_version: "encryptionkey@0"
I20260806 04:51:34.128285 16381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "5eeb73e54fad4f20b7aa7b82b5e970db"
format_stamp: "Formatted at 2026-08-06 04:51:34 on dist-test-slave-1x4g"
server_key: "a1dff734c998098b693eb8772b336ee4"
server_key_iv: "7e8da313987fd596cf2a682993a95f49"
server_key_version: "encryptionkey@0"
I20260806 04:51:34.133131 16381 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260806 04:51:34.135967 16396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:34.137259 16381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:51:34.137435 16381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "5eeb73e54fad4f20b7aa7b82b5e970db"
format_stamp: "Formatted at 2026-08-06 04:51:34 on dist-test-slave-1x4g"
server_key: "a1dff734c998098b693eb8772b336ee4"
server_key_iv: "7e8da313987fd596cf2a682993a95f49"
server_key_version: "encryptionkey@0"
I20260806 04:51:34.137630 16381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:34.160506 16381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:34.164043 16381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:34.164316 16381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:34.181967 16381 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:35439
I20260806 04:51:34.181981 16448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:35439 every 8 connection(s)
I20260806 04:51:34.183210 16381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:51:34.186360 16449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:34.188604 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 16381
I20260806 04:51:34.188854 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:51:34.189625 13906 external_mini_cluster.cc:1428] Setting key 8bf5dd1ee3b223a14314925d011944ce
I20260806 04:51:34.197768 16449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db: Bootstrap starting.
I20260806 04:51:34.200595 16449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:34.201614 16449 log.cc:824] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:51:34 dist-test-slave-1x4g krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:34.204082 16449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db: No bootstrap required, opened a new log
I20260806 04:51:34.212371 16452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:34.191715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.13.148.190:35439)
0806 04:51:34.192473 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:34.192483 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:34.192520 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:51:34.194089 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:34.194108 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:34.194473 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:34.194773 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:34.196322 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:34.197774 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:34.199178 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:34.199801 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:34.199984 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:34.203988 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:34.203998 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:34.204014 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:34.204057 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:34.207897 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:34.208536 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:34.208546 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:34.208556 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:34.208769 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:34.209142 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:34.209146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:34.209150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:34.209856 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:34.210255 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:34.210576 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 04:51:34.211049 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":357,"threads_started":1}
I20260806 04:51:34.214423 16449 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eeb73e54fad4f20b7aa7b82b5e970db" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 35439 } }
I20260806 04:51:34.214762 16449 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:34.214871 16449 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eeb73e54fad4f20b7aa7b82b5e970db, State: Initialized, Role: FOLLOWER
I20260806 04:51:34.215410 16449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eeb73e54fad4f20b7aa7b82b5e970db" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 35439 } }
I20260806 04:51:34.215668 16449 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:51:34.215804 16449 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:51:34.215975 16449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:34.218232 16449 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eeb73e54fad4f20b7aa7b82b5e970db" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 35439 } }
I20260806 04:51:34.218772 16449 leader_election.cc:302] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eeb73e54fad4f20b7aa7b82b5e970db; no voters:
I20260806 04:51:34.219401 16449 leader_election.cc:288] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:51:34.219544 16454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:34.220587 16454 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [term 1 LEADER]: Becoming Leader. State: Replica: 5eeb73e54fad4f20b7aa7b82b5e970db, State: Running, Role: LEADER
I20260806 04:51:34.221061 16454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eeb73e54fad4f20b7aa7b82b5e970db" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 35439 } }
I20260806 04:51:34.221513 16449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:51:34.223990 16456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eeb73e54fad4f20b7aa7b82b5e970db. Latest consensus state: current_term: 1 leader_uuid: "5eeb73e54fad4f20b7aa7b82b5e970db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eeb73e54fad4f20b7aa7b82b5e970db" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 35439 } } }
I20260806 04:51:34.224476 16456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:34.224504 16455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5eeb73e54fad4f20b7aa7b82b5e970db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eeb73e54fad4f20b7aa7b82b5e970db" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 35439 } } }
I20260806 04:51:34.224911 16455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:34.230108 16461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:51:34.234943 16461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:51:34.243994 16461 catalog_manager.cc:1347] Generated new cluster ID: c2b208eaa6774cab9d8804913da56a8b
I20260806 04:51:34.244100 16461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:51:34.271728 16461 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:51:34.273448 16461 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:51:34.283289 16461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5eeb73e54fad4f20b7aa7b82b5e970db: Generated new TSK 0
I20260806 04:51:34.284235 16461 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:51:34.334362 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:35439
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:42657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:34.627341 16477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:34.627566 16477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:34.627733 16477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:34.634605 16477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:34.634719 16477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:34.634904 16477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:51:34.643764 16477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:42657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:35439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:34.644840 16477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:34.645969 16477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:34.657987 16482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:34.658025 16485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:34.657935 16483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:34.658986 16477 server_base.cc:1034] running on GCE node
I20260806 04:51:34.659538 16477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:34.660367 16477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:34.661594 16477 hybrid_clock.cc:648] HybridClock initialized: now 1785991894661553 us; error 46 us; skew 500 ppm
I20260806 04:51:34.661873 16477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:34 dist-test-slave-1x4g krb5kdc[16367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991894, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:34.664657 16477 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:51:34.665936 16477 webserver.cc:460] Webserver started at http://127.13.148.129:43845/ using document root <none> and password file <none>
I20260806 04:51:34.666550 16477 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:34.666682 16477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:34.666965 16477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:34.669764 16477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "0453a1b2fc2e406fa13b187fe5272864"
format_stamp: "Formatted at 2026-08-06 04:51:34 on dist-test-slave-1x4g"
server_key: "9736dfbaef007e290f5f6804015fd03b"
server_key_iv: "b0ecec7cd0b5471920055e188865a0d6"
server_key_version: "encryptionkey@0"
I20260806 04:51:34.670477 16477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "0453a1b2fc2e406fa13b187fe5272864"
format_stamp: "Formatted at 2026-08-06 04:51:34 on dist-test-slave-1x4g"
server_key: "9736dfbaef007e290f5f6804015fd03b"
server_key_iv: "b0ecec7cd0b5471920055e188865a0d6"
server_key_version: "encryptionkey@0"
I20260806 04:51:34.674916 16477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:51:34.678037 16492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:34.679250 16477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:51:34.679407 16477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "0453a1b2fc2e406fa13b187fe5272864"
format_stamp: "Formatted at 2026-08-06 04:51:34 on dist-test-slave-1x4g"
server_key: "9736dfbaef007e290f5f6804015fd03b"
server_key_iv: "b0ecec7cd0b5471920055e188865a0d6"
server_key_version: "encryptionkey@0"
I20260806 04:51:34.679638 16477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:34.705385 16477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:34.709028 16477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:34.709306 16477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:34.710140 16477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:34.711441 16477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:34.711508 16477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:34.711654 16477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:34.711704 16477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:34.742571 16477 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:44645
I20260806 04:51:34.742642 16607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:44645 every 8 connection(s)
I20260806 04:51:34.743644 16477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:51:34.749507 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 16477
I20260806 04:51:34.749723 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:51:34.750293 13906 external_mini_cluster.cc:1428] Setting key bd1cf590c52a54032575422e2b75fa11
Aug 06 04:51:34 dist-test-slave-1x4g krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991894, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:34.762693 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:34.745943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:59941 (local address 127.13.148.190:35439)
0806 04:51:34.746311 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:34.746315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:34.747179 (+ 864us) server_negotiation.cc:407] Connection header received
0806 04:51:34.748929 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:34.748935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:34.749029 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:34.749154 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:34.750698 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:34.751482 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:34.752991 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:34.753761 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:34.753902 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:34.757021 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:34.757030 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:34.757037 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:34.757079 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:34.759715 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:34.760631 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:34.760634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:34.760636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:34.760759 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:34.761268 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:34.761271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:34.761273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:34.761507 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:34.761607 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:34.762343 (+ 736us) server_negotiation.cc:299] Negotiation successful
0806 04:51:34.762474 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":182,"threads_started":1}
I20260806 04:51:34.763978 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:34.746452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:35439 (local address 127.13.148.129:59941)
0806 04:51:34.747037 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:34.747071 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:34.748613 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:34.749387 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:34.749395 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:34.750004 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:34.750548 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:34.750564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:34.751616 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:34.751620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:34.752862 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:34.752872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:34.753974 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:34.753979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:34.754155 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:34.754786 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:34.754805 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:34.756766 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:34.759843 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:34.759854 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:34.759870 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:34.760517 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:34.760903 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:34.760910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:34.760916 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:34.761142 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:34.761624 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:34.761634 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:34.762018 (+ 384us) client_negotiation.cc:769] Sending connection context
0806 04:51:34.762273 (+ 255us) client_negotiation.cc:240] Negotiation successful
0806 04:51:34.762741 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":224,"threads_started":1}
I20260806 04:51:34.765462 16608 heartbeater.cc:347] Connected to a master server at 127.13.148.190:35439
I20260806 04:51:34.765722 16608 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:34.766398 16608 heartbeater.cc:511] Master 127.13.148.190:35439 requested a full tablet report, sending...
I20260806 04:51:34.768330 16413 ts_manager.cc:194] Registered new tserver with Master: 0453a1b2fc2e406fa13b187fe5272864 (127.13.148.129:44645)
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
I20260806 04:51:34.769807 16413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:59941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:51:34.792035 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:35439
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:42657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:35.083598 16614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:35.083828 16614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:35.083992 16614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:35.090571 16614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:35.090656 16614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:35.090804 16614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:51:35.099591 16614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:42657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:35439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:35.100667 16614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:35.101857 16614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:35.113904 16620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:35.113926 16621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:35.113945 16623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:35.114663 16614 server_base.cc:1034] running on GCE node
I20260806 04:51:35.115197 16614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:35.115936 16614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:35.117120 16614 hybrid_clock.cc:648] HybridClock initialized: now 1785991895117077 us; error 32 us; skew 500 ppm
I20260806 04:51:35.117372 16614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:35 dist-test-slave-1x4g krb5kdc[16367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991895, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:35.120213 16614 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:51:35.121578 16614 webserver.cc:460] Webserver started at http://127.13.148.130:45421/ using document root <none> and password file <none>
I20260806 04:51:35.122191 16614 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:35.122305 16614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:35.122624 16614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:35.125214 16614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "614240892a324da8b1f5a22bd16d3c15"
format_stamp: "Formatted at 2026-08-06 04:51:35 on dist-test-slave-1x4g"
server_key: "061d234a5e0af9fdf4ad56476dec4a04"
server_key_iv: "9bd77b421fb06186d92c0e11055273d6"
server_key_version: "encryptionkey@0"
I20260806 04:51:35.125939 16614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "614240892a324da8b1f5a22bd16d3c15"
format_stamp: "Formatted at 2026-08-06 04:51:35 on dist-test-slave-1x4g"
server_key: "061d234a5e0af9fdf4ad56476dec4a04"
server_key_iv: "9bd77b421fb06186d92c0e11055273d6"
server_key_version: "encryptionkey@0"
I20260806 04:51:35.130426 16614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:51:35.133347 16630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:35.134447 16614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:51:35.134606 16614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "614240892a324da8b1f5a22bd16d3c15"
format_stamp: "Formatted at 2026-08-06 04:51:35 on dist-test-slave-1x4g"
server_key: "061d234a5e0af9fdf4ad56476dec4a04"
server_key_iv: "9bd77b421fb06186d92c0e11055273d6"
server_key_version: "encryptionkey@0"
I20260806 04:51:35.134764 16614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:35.147941 16614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:35.151330 16614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:35.151569 16614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:35.152357 16614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:35.153550 16614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:35.153616 16614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:35.153761 16614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:35.153813 16614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:35.185067 16614 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:39899
I20260806 04:51:35.185138 16744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:39899 every 8 connection(s)
I20260806 04:51:35.186388 16614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:51:35.196442 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 16614
I20260806 04:51:35.196655 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:51:35.197245 13906 external_mini_cluster.cc:1428] Setting key 2c3709607420d3d7de877c6d47c6602e
Aug 06 04:51:35 dist-test-slave-1x4g krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991895, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:35.206557 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:35.188879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:55229 (local address 127.13.148.190:35439)
0806 04:51:35.189054 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:35.189059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:35.190147 (+ 1088us) server_negotiation.cc:407] Connection header received
0806 04:51:35.191887 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:35.191893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:35.191986 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:35.192112 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:35.193585 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:35.194337 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:35.196015 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:35.196461 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:35.196653 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:35.200869 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:35.200873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:35.200876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:35.200902 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:35.203414 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:35.204307 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:35.204310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:35.204313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:35.204435 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:35.204918 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:35.204921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:35.204923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:35.205264 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:35.205402 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:35.205961 (+ 559us) server_negotiation.cc:299] Negotiation successful
0806 04:51:35.206196 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:51:35.207047 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:35.189337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:35439 (local address 127.13.148.130:55229)
0806 04:51:35.190007 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:35.190040 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:35.191591 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:35.192326 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:35.192333 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:35.192913 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:35.193421 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:35.193438 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:35.194624 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:35.194628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:35.195869 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:35.195881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:35.196765 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:35.196773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:35.197084 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:35.197853 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:35.197882 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:35.200631 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:35.203574 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:35.203585 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:35.203601 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:35.204177 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:35.204538 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:35.204545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:35.204551 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:35.204815 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:35.205378 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:35.205383 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:35.205644 (+ 261us) client_negotiation.cc:769] Sending connection context
0806 04:51:35.205826 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 04:51:35.206132 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":282,"threads_started":1}
I20260806 04:51:35.208376 16745 heartbeater.cc:347] Connected to a master server at 127.13.148.190:35439
I20260806 04:51:35.208683 16745 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:35.209425 16745 heartbeater.cc:511] Master 127.13.148.190:35439 requested a full tablet report, sending...
I20260806 04:51:35.210728 16413 ts_manager.cc:194] Registered new tserver with Master: 614240892a324da8b1f5a22bd16d3c15 (127.13.148.130:39899)
I20260806 04:51:35.211459 16413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:55229
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
I20260806 04:51:35.237735 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:35439
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:42657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:35.531632 16750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:35.531860 16750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:35.532011 16750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:35.538340 16750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:35.538456 16750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:35.538653 16750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:51:35.547309 16750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:42657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:35439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:35.548460 16750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:35.549858 16750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:35.562176 16759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:35.562141 16755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:35.562212 16756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:35.562844 16750 server_base.cc:1034] running on GCE node
I20260806 04:51:35.563381 16750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:35.564178 16750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:35.565376 16750 hybrid_clock.cc:648] HybridClock initialized: now 1785991895565343 us; error 46 us; skew 500 ppm
I20260806 04:51:35.565650 16750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:35 dist-test-slave-1x4g krb5kdc[16367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991895, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:35.573441 16750 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:51:35.574724 16750 webserver.cc:460] Webserver started at http://127.13.148.131:37791/ using document root <none> and password file <none>
I20260806 04:51:35.575321 16750 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:35.575449 16750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:35.575729 16750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:35.578485 16750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "d845e1b37ae143439d3027a54cd252f3"
format_stamp: "Formatted at 2026-08-06 04:51:35 on dist-test-slave-1x4g"
server_key: "02dee5e7c63cef1c48452a76a33215e0"
server_key_iv: "cb2456e111a39aa8fe08f30f3db588e2"
server_key_version: "encryptionkey@0"
I20260806 04:51:35.579272 16750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "d845e1b37ae143439d3027a54cd252f3"
format_stamp: "Formatted at 2026-08-06 04:51:35 on dist-test-slave-1x4g"
server_key: "02dee5e7c63cef1c48452a76a33215e0"
server_key_iv: "cb2456e111a39aa8fe08f30f3db588e2"
server_key_version: "encryptionkey@0"
I20260806 04:51:35.584120 16750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260806 04:51:35.587132 16766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:35.588286 16750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:51:35.588462 16750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "d845e1b37ae143439d3027a54cd252f3"
format_stamp: "Formatted at 2026-08-06 04:51:35 on dist-test-slave-1x4g"
server_key: "02dee5e7c63cef1c48452a76a33215e0"
server_key_iv: "cb2456e111a39aa8fe08f30f3db588e2"
server_key_version: "encryptionkey@0"
I20260806 04:51:35.588657 16750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:35.599777 16750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:35.603037 16750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:35.603286 16750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:35.604091 16750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:35.605271 16750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:35.605365 16750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:35.605527 16750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:35.605611 16750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:35.636427 16750 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:41775
I20260806 04:51:35.636463 16879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:41775 every 8 connection(s)
I20260806 04:51:35.637908 16750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:51:35.642968 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 16750
I20260806 04:51:35.643162 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:51:35.643651 13906 external_mini_cluster.cc:1428] Setting key 28f4cfcdec16c536626f005c89183fca
Aug 06 04:51:35 dist-test-slave-1x4g krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991895, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:35.657657 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:35.640534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:42263 (local address 127.13.148.190:35439)
0806 04:51:35.640706 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:35.640710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:35.641991 (+ 1281us) server_negotiation.cc:407] Connection header received
0806 04:51:35.644327 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:35.644332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:35.644425 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:35.644538 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:35.646108 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:35.646812 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:35.648318 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:35.649085 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:35.649250 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:35.652509 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:35.652513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:35.652516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:35.652543 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:35.654835 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:35.655460 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:35.655463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:35.655465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:35.655582 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:35.655984 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:35.655987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:35.655989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:35.656237 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:35.656334 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:35.657100 (+ 766us) server_negotiation.cc:299] Negotiation successful
0806 04:51:35.657299 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 04:51:35.658758 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:35.641062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:35439 (local address 127.13.148.131:42263)
0806 04:51:35.641778 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:35.641833 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:35.644031 (+ 2198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:35.644766 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:35.644808 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:35.645389 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:35.645950 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:35.645965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:35.646949 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:35.646955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:35.648140 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:35.648152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:35.649349 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:35.649355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:35.649667 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:35.650335 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:35.650355 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:35.652334 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:35.654954 (+ 2620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:35.654961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:35.654971 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:35.655353 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:35.655697 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:35.655700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:35.655702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:35.655875 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:35.656331 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:35.656336 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:35.656621 (+ 285us) client_negotiation.cc:769] Sending connection context
0806 04:51:35.656942 (+ 321us) client_negotiation.cc:240] Negotiation successful
0806 04:51:35.657437 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":266,"threads_started":1}
I20260806 04:51:35.660023 16880 heartbeater.cc:347] Connected to a master server at 127.13.148.190:35439
I20260806 04:51:35.660298 16880 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:35.661023 16880 heartbeater.cc:511] Master 127.13.148.190:35439 requested a full tablet report, sending...
I20260806 04:51:35.662281 16413 ts_manager.cc:194] Registered new tserver with Master: d845e1b37ae143439d3027a54cd252f3 (127.13.148.131:41775)
I20260806 04:51:35.663014 16413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:42263
I20260806 04:51:35.670369 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 04:51:35.700984 16610 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:51:35.698615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47268 (local address 127.13.148.190:35439)
0806 04:51:35.698805 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:35.698809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:35.699277 (+ 468us) server_negotiation.cc:407] Connection header received
0806 04:51:35.699423 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:35.699427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:35.699493 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:35.699591 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:35.700690 (+ 1099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47268: BlockingRecv error: recv got EOF from 127.0.0.1:47268 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 04:51:35.704285 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 16477
W20260806 04:51:35.706771 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 16477: Not authorized: Client connection negotiation failed: client connection to 127.13.148.129:44645: server requires authentication, but client does not have Kerberos credentials available
I20260806 04:51:35.706933 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 16477
I20260806 04:51:35.724378 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 16614
W20260806 04:51:35.726516 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 16614: Not authorized: Client connection negotiation failed: client connection to 127.13.148.130:39899: server requires authentication, but client does not have Kerberos credentials available
I20260806 04:51:35.726665 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 16614
W20260806 04:51:35.726815 16892 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:51:35.725130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60488 (local address 127.13.148.130:39899)
0806 04:51:35.725582 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:35.725591 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:35.725614 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:51:35.725667 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:35.725673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:35.725865 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:35.725978 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:35.726535 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60488: BlockingRecv error: recv got EOF from 127.0.0.1:60488 (error 108)
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":226,"threads_started":1}
I20260806 04:51:35.743016 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 16750
W20260806 04:51:35.745421 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 16750: Not authorized: Client connection negotiation failed: client connection to 127.13.148.131:41775: server requires authentication, but client does not have Kerberos credentials available
I20260806 04:51:35.745556 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 16750
I20260806 04:51:35.761072 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 16381
I20260806 04:51:35.762121 16444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:35.856218 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 16381
2026-08-06T04:51:35Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3246 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:51:35 dist-test-slave-1x4g krb5kdc[16897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:51:35 dist-test-slave-1x4g krb5kdc[16897](info): set up 2 sockets
Aug 06 04:51:35 dist-test-slave-1x4g krb5kdc[16897](info): commencing operation
krb5kdc: starting...
W20260806 04:51:36.869704 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:51:36 dist-test-slave-1x4g krb5kdc[16897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:51:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:51:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:51:37.040699 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:45977
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:41475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:45977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:37.332353 16911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:37.332613 16911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:37.332695 16911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:37.339366 16911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:51:37.339462 16911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:37.339529 16911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:37.339600 16911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:51:37.339670 16911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:51:37.348369 16911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:41475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:45977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:45977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:37.349542 16911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:37.350701 16911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:37.361750 16916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:37.361799 16917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:37.361836 16919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:37.362577 16911 server_base.cc:1034] running on GCE node
I20260806 04:51:37.363317 16911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:37.364836 16911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:37.366034 16911 hybrid_clock.cc:648] HybridClock initialized: now 1785991897366006 us; error 28 us; skew 500 ppm
I20260806 04:51:37.366355 16911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:37 dist-test-slave-1x4g krb5kdc[16897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991897, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:37.369254 16911 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:51:37.370641 16911 webserver.cc:460] Webserver started at http://127.13.148.190:35419/ using document root <none> and password file <none>
I20260806 04:51:37.371294 16911 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:37.371417 16911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:37.371765 16911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:37.374487 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "0619413b0e2a4cadbd0b16ccfcec05f2"
format_stamp: "Formatted at 2026-08-06 04:51:37 on dist-test-slave-1x4g"
server_key: "8a7b07817bf295a6e18b7ab8423943ec"
server_key_iv: "04e7b2d2b3056423555eab54ef9f0db0"
server_key_version: "encryptionkey@0"
I20260806 04:51:37.375229 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "0619413b0e2a4cadbd0b16ccfcec05f2"
format_stamp: "Formatted at 2026-08-06 04:51:37 on dist-test-slave-1x4g"
server_key: "8a7b07817bf295a6e18b7ab8423943ec"
server_key_iv: "04e7b2d2b3056423555eab54ef9f0db0"
server_key_version: "encryptionkey@0"
I20260806 04:51:37.379978 16911 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260806 04:51:37.383011 16926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:37.384155 16911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:51:37.384320 16911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "0619413b0e2a4cadbd0b16ccfcec05f2"
format_stamp: "Formatted at 2026-08-06 04:51:37 on dist-test-slave-1x4g"
server_key: "8a7b07817bf295a6e18b7ab8423943ec"
server_key_iv: "04e7b2d2b3056423555eab54ef9f0db0"
server_key_version: "encryptionkey@0"
I20260806 04:51:37.384527 16911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:37.403196 16911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:37.406538 16911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:37.406728 16911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:37.424003 16911 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:45977
I20260806 04:51:37.424043 16980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:45977 every 8 connection(s)
I20260806 04:51:37.425257 16911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:51:37.428679 16981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:37.435596 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 16911
I20260806 04:51:37.435832 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:51:37.436443 13906 external_mini_cluster.cc:1428] Setting key a0512dab51d8bf8ccba15092681369c6
I20260806 04:51:37.438459 16981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2: Bootstrap starting.
I20260806 04:51:37.441574 16981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:37.442826 16981 log.cc:824] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:37.445642 16981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2: No bootstrap required, opened a new log
Aug 06 04:51:37 dist-test-slave-1x4g krb5kdc[16897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:37.453649 16981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0619413b0e2a4cadbd0b16ccfcec05f2" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45977 } }
I20260806 04:51:37.453986 16981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:37.454097 16981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0619413b0e2a4cadbd0b16ccfcec05f2, State: Initialized, Role: FOLLOWER
I20260806 04:51:37.454582 16981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0619413b0e2a4cadbd0b16ccfcec05f2" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45977 } }
I20260806 04:51:37.454797 16981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:51:37.454921 16981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:51:37.455072 16981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:37.457127 16981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0619413b0e2a4cadbd0b16ccfcec05f2" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45977 } }
I20260806 04:51:37.457669 16981 leader_election.cc:302] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0619413b0e2a4cadbd0b16ccfcec05f2; no voters:
I20260806 04:51:37.458256 16981 leader_election.cc:288] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:51:37.458400 16986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:37.458796 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:37.438314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.13.148.190:45977)
0806 04:51:37.439198 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:37.439209 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:37.439249 (+ 40us) server_negotiation.cc:407] Connection header received
0806 04:51:37.440929 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:37.440956 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:37.441292 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:37.441590 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:37.443383 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:37.444899 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:37.446456 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:37.447048 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:37.447172 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:37.451569 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:37.451577 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:37.451590 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:37.451626 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:37.454597 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:37.455290 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:37.455300 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:37.455309 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:37.455469 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:37.455813 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:37.455818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:37.455821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:37.456286 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:37.456567 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:37.457043 (+ 476us) server_negotiation.cc:299] Negotiation successful
0806 04:51:37.457428 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":415,"threads_started":1}
I20260806 04:51:37.459625 16986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [term 1 LEADER]: Becoming Leader. State: Replica: 0619413b0e2a4cadbd0b16ccfcec05f2, State: Running, Role: LEADER
I20260806 04:51:37.460172 16986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0619413b0e2a4cadbd0b16ccfcec05f2" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45977 } }
I20260806 04:51:37.460300 16981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:51:37.462692 16987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0619413b0e2a4cadbd0b16ccfcec05f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0619413b0e2a4cadbd0b16ccfcec05f2" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45977 } } }
I20260806 04:51:37.463032 16987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:37.463683 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:51:37.466826 16991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:51:37.466709 16988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0619413b0e2a4cadbd0b16ccfcec05f2. Latest consensus state: current_term: 1 leader_uuid: "0619413b0e2a4cadbd0b16ccfcec05f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0619413b0e2a4cadbd0b16ccfcec05f2" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45977 } } }
I20260806 04:51:37.466995 16988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:37.471838 16991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:51:37.492692 16991 catalog_manager.cc:1347] Generated new cluster ID: 8ef48869a52b42a1aa056ed898c33501
I20260806 04:51:37.492815 16991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:51:37.521736 16991 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:51:37.523507 16991 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:51:37.532544 16991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0619413b0e2a4cadbd0b16ccfcec05f2: Generated new TSK 0
I20260806 04:51:37.532966 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:37.525968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.13.148.190:45977)
0806 04:51:37.526150 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:37.526158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:37.526675 (+ 517us) server_negotiation.cc:407] Connection header received
0806 04:51:37.526861 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:37.526865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:37.526947 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:37.527043 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:37.528010 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:37.528742 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:37.529977 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:37.530392 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:37.530524 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:37.531613 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:37.531619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:37.531633 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:37.532170 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:37.532224 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:37.532244 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:37.532357 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:37.532543 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 04:51:37.532643 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:51:37.533490 16991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:51:37.546172 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 16911
I20260806 04:51:37.547212 16976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:37.630272 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 16911
2026-08-06T04:51:37Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1767 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:51:37 dist-test-slave-1x4g krb5kdc[17015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:51:37 dist-test-slave-1x4g krb5kdc[17015](info): set up 2 sockets
Aug 06 04:51:37 dist-test-slave-1x4g krb5kdc[17015](info): commencing operation
krb5kdc: starting...
W20260806 04:51:38.644923 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260806 04:51:38.822170 13906 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.177s user 0.000s sys 0.006s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260806 04:51:38.946002 13906 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.124s user 0.001s sys 0.006s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260806 04:51:39.139659 13906 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.193s user 0.001s sys 0.006s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/test-user.keytab.
W20260806 04:51:39.840929 13906 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.701s user 0.002s sys 0.005s
Aug 06 04:51:39 dist-test-slave-1x4g krb5kdc[17015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:51:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:51:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
W20260806 04:51:40.255028 13906 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.13.148.190: real 0.362s user 0.004s sys 0.005s
I20260806 04:51:40.255928 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:44369
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:44369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:40.557828 17030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:40.558089 17030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:40.558164 17030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:40.565096 17030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:51:40.565197 17030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:40.565281 17030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:40.565366 17030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:51:40.565440 17030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:51:40.574183 17030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:44369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:44369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:40.575464 17030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:40.576651 17030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:40.587410 17036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:40.587306 17035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:40.587450 17038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:40.588287 17030 server_base.cc:1034] running on GCE node
I20260806 04:51:40.589099 17030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:40.590437 17030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:40.591663 17030 hybrid_clock.cc:648] HybridClock initialized: now 1785991900591620 us; error 51 us; skew 500 ppm
I20260806 04:51:40.592012 17030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:40 dist-test-slave-1x4g krb5kdc[17015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991900, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:40.599133 17030 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:51:40.600616 17030 webserver.cc:460] Webserver started at http://127.13.148.190:34361/ using document root <none> and password file <none>
I20260806 04:51:40.601274 17030 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:40.601408 17030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:40.601715 17030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:40.604450 17030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "e79c6f12a22b4456b80ed3b13cf869d9"
format_stamp: "Formatted at 2026-08-06 04:51:40 on dist-test-slave-1x4g"
server_key: "518ffb28ba2167a18ab98147d8e780da"
server_key_iv: "42456cd4f508ca1c11351b8004875769"
server_key_version: "encryptionkey@0"
I20260806 04:51:40.605193 17030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "e79c6f12a22b4456b80ed3b13cf869d9"
format_stamp: "Formatted at 2026-08-06 04:51:40 on dist-test-slave-1x4g"
server_key: "518ffb28ba2167a18ab98147d8e780da"
server_key_iv: "42456cd4f508ca1c11351b8004875769"
server_key_version: "encryptionkey@0"
I20260806 04:51:40.609978 17030 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:51:40.612989 17045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:40.614221 17030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:51:40.614396 17030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "e79c6f12a22b4456b80ed3b13cf869d9"
format_stamp: "Formatted at 2026-08-06 04:51:40 on dist-test-slave-1x4g"
server_key: "518ffb28ba2167a18ab98147d8e780da"
server_key_iv: "42456cd4f508ca1c11351b8004875769"
server_key_version: "encryptionkey@0"
I20260806 04:51:40.614574 17030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:40.647480 17030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:40.651228 17030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:40.651494 17030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:40.669062 17030 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:44369
I20260806 04:51:40.669106 17097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:44369 every 8 connection(s)
I20260806 04:51:40.670418 17030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:51:40.673991 17098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:40.674379 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 17030
I20260806 04:51:40.674590 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:51:40.675333 13906 external_mini_cluster.cc:1428] Setting key 7ba5d102900b4d8ba093ab6df2cdaaf0
I20260806 04:51:40.687037 17098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9: Bootstrap starting.
Aug 06 04:51:40 dist-test-slave-1x4g krb5kdc[17015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:40.690624 17098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:40.691854 17098 log.cc:824] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:40.694543 17098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9: No bootstrap required, opened a new log
I20260806 04:51:40.697136 17101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:40.677316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35944 (local address 127.13.148.190:44369)
0806 04:51:40.678116 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:40.678126 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:40.678163 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:51:40.680125 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:40.680146 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:40.680496 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:40.680855 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:40.682340 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:40.683405 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:40.685008 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:40.685596 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:40.685713 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:40.690220 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:40.690228 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:40.690241 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:40.690291 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:40.693372 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:40.693983 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:40.693995 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:40.694000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:40.694155 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:40.694530 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:40.694537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:40.694539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:40.695057 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:40.695346 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:40.695729 (+ 383us) server_negotiation.cc:299] Negotiation successful
0806 04:51:40.696106 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":368,"threads_started":1}
I20260806 04:51:40.701917 17098 raft_consensus.cc:348] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79c6f12a22b4456b80ed3b13cf869d9" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44369 } }
I20260806 04:51:40.702245 17098 raft_consensus.cc:374] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:40.702338 17098 raft_consensus.cc:729] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79c6f12a22b4456b80ed3b13cf869d9, State: Initialized, Role: FOLLOWER
I20260806 04:51:40.702919 17098 consensus_queue.cc:260] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79c6f12a22b4456b80ed3b13cf869d9" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44369 } }
I20260806 04:51:40.703176 17098 raft_consensus.cc:388] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:51:40.703276 17098 raft_consensus.cc:482] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:51:40.703406 17098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:40.705474 17098 raft_consensus.cc:504] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79c6f12a22b4456b80ed3b13cf869d9" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44369 } }
I20260806 04:51:40.705878 17098 leader_election.cc:302] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e79c6f12a22b4456b80ed3b13cf869d9; no voters:
I20260806 04:51:40.706394 17098 leader_election.cc:288] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:51:40.706485 17103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:40.707548 17103 raft_consensus.cc:686] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [term 1 LEADER]: Becoming Leader. State: Replica: e79c6f12a22b4456b80ed3b13cf869d9, State: Running, Role: LEADER
I20260806 04:51:40.707993 17103 consensus_queue.cc:237] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79c6f12a22b4456b80ed3b13cf869d9" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44369 } }
I20260806 04:51:40.708369 17098 sys_catalog.cc:564] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:51:40.710840 17106 sys_catalog.cc:455] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e79c6f12a22b4456b80ed3b13cf869d9. Latest consensus state: current_term: 1 leader_uuid: "e79c6f12a22b4456b80ed3b13cf869d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79c6f12a22b4456b80ed3b13cf869d9" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44369 } } }
I20260806 04:51:40.711247 17106 sys_catalog.cc:458] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:40.711803 17104 sys_catalog.cc:455] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e79c6f12a22b4456b80ed3b13cf869d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79c6f12a22b4456b80ed3b13cf869d9" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44369 } } }
I20260806 04:51:40.712155 17104 sys_catalog.cc:458] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:40.712823 17109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:51:40.718040 17109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:51:40.726977 17109 catalog_manager.cc:1347] Generated new cluster ID: e11594ad2fd746059a01ac6dc990b621
I20260806 04:51:40.727092 17109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:51:40.751048 17109 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:51:40.752363 17109 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:51:40.758603 17109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e79c6f12a22b4456b80ed3b13cf869d9: Generated new TSK 0
I20260806 04:51:40.759364 17109 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:51:40.810665 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44369
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:41.129873 17125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:41.130123 17125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:41.130206 17125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:41.136319 17125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:41.136406 17125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:41.136497 17125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:51:41.144802 17125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:41.145805 17125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:41.146920 17125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:41.158612 17133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:41.158687 17130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:41.158660 17131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:41.159749 17125 server_base.cc:1034] running on GCE node
I20260806 04:51:41.160306 17125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:41.161154 17125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:41.162335 17125 hybrid_clock.cc:648] HybridClock initialized: now 1785991901162284 us; error 42 us; skew 500 ppm
I20260806 04:51:41.162626 17125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:41 dist-test-slave-1x4g krb5kdc[17015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991901, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:41.165494 17125 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:51:41.166713 17125 webserver.cc:460] Webserver started at http://127.13.148.129:44995/ using document root <none> and password file <none>
I20260806 04:51:41.167254 17125 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:41.167341 17125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:41.167562 17125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:41.170308 17125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "c7814bbe4e834c7c85c8f6cb8d898e3e"
format_stamp: "Formatted at 2026-08-06 04:51:41 on dist-test-slave-1x4g"
server_key: "4df003d6b91b208b363fc7c35c2b523b"
server_key_iv: "642f28a66ddc6382a06f2e3171453ea2"
server_key_version: "encryptionkey@0"
I20260806 04:51:41.170967 17125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "c7814bbe4e834c7c85c8f6cb8d898e3e"
format_stamp: "Formatted at 2026-08-06 04:51:41 on dist-test-slave-1x4g"
server_key: "4df003d6b91b208b363fc7c35c2b523b"
server_key_iv: "642f28a66ddc6382a06f2e3171453ea2"
server_key_version: "encryptionkey@0"
I20260806 04:51:41.175436 17125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260806 04:51:41.178699 17140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:41.179960 17125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:51:41.180119 17125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "c7814bbe4e834c7c85c8f6cb8d898e3e"
format_stamp: "Formatted at 2026-08-06 04:51:41 on dist-test-slave-1x4g"
server_key: "4df003d6b91b208b363fc7c35c2b523b"
server_key_iv: "642f28a66ddc6382a06f2e3171453ea2"
server_key_version: "encryptionkey@0"
I20260806 04:51:41.180285 17125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:41.200289 17125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:41.203825 17125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:41.204107 17125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:41.205013 17125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:41.206172 17125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:41.206243 17125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:41.206393 17125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:41.206451 17125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:41.238122 17125 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:35487
I20260806 04:51:41.238163 17256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:35487 every 8 connection(s)
I20260806 04:51:41.239416 17125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:51:41.249217 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 17125
I20260806 04:51:41.249445 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:51:41.250180 13906 external_mini_cluster.cc:1428] Setting key 67da29fc93310aa11c15ede976017811
Aug 06 04:51:41 dist-test-slave-1x4g krb5kdc[17015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991901, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:41.258680 17259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:41.241772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:35721 (local address 127.13.148.190:44369)
0806 04:51:41.242199 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:41.242207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:41.243055 (+ 848us) server_negotiation.cc:407] Connection header received
0806 04:51:41.244626 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:41.244629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:41.244691 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:41.244815 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:41.246064 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:41.246683 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:41.247977 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:41.248315 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:41.248398 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:41.252136 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:41.252143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:41.252147 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:41.252180 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:41.255527 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:41.256157 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:41.256164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:41.256170 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:41.256331 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:41.256725 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:41.256732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:41.256738 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:41.257168 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:41.257322 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:41.257937 (+ 615us) server_negotiation.cc:299] Negotiation successful
0806 04:51:41.258172 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":210,"threads_started":1}
I20260806 04:51:41.259516 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:41.242272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:44369 (local address 127.13.148.129:35721)
0806 04:51:41.242902 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:41.242952 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:41.244402 (+ 1450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:41.245011 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:41.245019 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:41.245441 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:41.245921 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:41.245950 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:41.246798 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:41.246801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:41.247861 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:41.247871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:41.248501 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:41.248506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:41.248685 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:41.249449 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:41.249470 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:41.251913 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:41.255654 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:41.255660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:41.255670 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:41.256053 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:41.256430 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:41.256433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:41.256435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:41.256604 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:41.257289 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:41.257295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:41.257590 (+ 295us) client_negotiation.cc:769] Sending connection context
0806 04:51:41.257806 (+ 216us) client_negotiation.cc:240] Negotiation successful
0806 04:51:41.258262 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":242,"threads_started":1}
I20260806 04:51:41.261214 17257 heartbeater.cc:347] Connected to a master server at 127.13.148.190:44369
I20260806 04:51:41.261487 17257 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:41.262120 17257 heartbeater.cc:511] Master 127.13.148.190:44369 requested a full tablet report, sending...
I20260806 04:51:41.264252 17062 ts_manager.cc:194] Registered new tserver with Master: c7814bbe4e834c7c85c8f6cb8d898e3e (127.13.148.129:35487)
I20260806 04:51:41.266256 17062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:35721
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:51:41.293968 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44369
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:41.586988 17263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:41.587225 17263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:41.587397 17263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:41.593688 17263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:41.593796 17263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:41.593966 17263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:51:41.602414 17263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:41.603488 17263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:41.604630 17263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:41.616901 17268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:41.616827 17271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:41.616946 17269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:41.617798 17263 server_base.cc:1034] running on GCE node
I20260806 04:51:41.618376 17263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:41.619344 17263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:41.620533 17263 hybrid_clock.cc:648] HybridClock initialized: now 1785991901620493 us; error 42 us; skew 500 ppm
I20260806 04:51:41.620853 17263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:41 dist-test-slave-1x4g krb5kdc[17015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991901, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:41.623874 17263 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:51:41.625284 17263 webserver.cc:460] Webserver started at http://127.13.148.130:33897/ using document root <none> and password file <none>
I20260806 04:51:41.625965 17263 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:41.626108 17263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:41.626448 17263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:41.629310 17263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "9cb0463c12ae44fe8a4345d7534a302c"
format_stamp: "Formatted at 2026-08-06 04:51:41 on dist-test-slave-1x4g"
server_key: "5ec65bc0d7dd639a71963ae2bc59bc70"
server_key_iv: "a4b39709f0fdf29b994b874fa405a763"
server_key_version: "encryptionkey@0"
I20260806 04:51:41.630065 17263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "9cb0463c12ae44fe8a4345d7534a302c"
format_stamp: "Formatted at 2026-08-06 04:51:41 on dist-test-slave-1x4g"
server_key: "5ec65bc0d7dd639a71963ae2bc59bc70"
server_key_iv: "a4b39709f0fdf29b994b874fa405a763"
server_key_version: "encryptionkey@0"
I20260806 04:51:41.634850 17263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:51:41.638011 17278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:41.639307 17263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:51:41.639480 17263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "9cb0463c12ae44fe8a4345d7534a302c"
format_stamp: "Formatted at 2026-08-06 04:51:41 on dist-test-slave-1x4g"
server_key: "5ec65bc0d7dd639a71963ae2bc59bc70"
server_key_iv: "a4b39709f0fdf29b994b874fa405a763"
server_key_version: "encryptionkey@0"
I20260806 04:51:41.639688 17263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:41.659399 17263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:41.663182 17263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:41.663501 17263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:41.664412 17263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:41.665693 17263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:41.665795 17263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:41.665930 17263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:41.666002 17263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:41.696854 17263 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:38221
I20260806 04:51:41.696887 17395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:38221 every 8 connection(s)
I20260806 04:51:41.698099 17263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:51:41.700297 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 17263
I20260806 04:51:41.700506 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:51:41.700971 13906 external_mini_cluster.cc:1428] Setting key 74ec71eafdf749b05bbc10c89673965a
Aug 06 04:51:41 dist-test-slave-1x4g krb5kdc[17015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991901, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:41.718295 17259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:41.700501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:44343 (local address 127.13.148.190:44369)
0806 04:51:41.700694 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:41.700699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:41.702008 (+ 1309us) server_negotiation.cc:407] Connection header received
0806 04:51:41.703708 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:41.703715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:41.703799 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:41.703904 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:41.705293 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:41.706332 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:41.707701 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:41.708377 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:41.708529 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:41.712611 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:41.712616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:41.712620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:41.712654 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:41.715315 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:41.715981 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:41.715991 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:41.715995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:41.716113 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:41.716512 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:41.716517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:41.716521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:41.716903 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:41.717060 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:41.717735 (+ 675us) server_negotiation.cc:299] Negotiation successful
0806 04:51:41.717998 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 04:51:41.719229 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:41.701098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:44369 (local address 127.13.148.130:44343)
0806 04:51:41.701862 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:41.701905 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:41.703466 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:41.704086 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:41.704093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:41.704550 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:41.705136 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:41.705153 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:41.706459 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:41.706463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:41.707564 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:41.707575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:41.708638 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:41.708648 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:41.709037 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:41.709771 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:41.709813 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:41.712402 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:41.715460 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:41.715468 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:41.715481 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:41.715860 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:41.716219 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:41.716223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:41.716227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:41.716395 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:41.717034 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:41.717041 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:41.717350 (+ 309us) client_negotiation.cc:769] Sending connection context
0806 04:51:41.717593 (+ 243us) client_negotiation.cc:240] Negotiation successful
0806 04:51:41.718129 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":295,"threads_started":1}
I20260806 04:51:41.720566 17396 heartbeater.cc:347] Connected to a master server at 127.13.148.190:44369
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
I20260806 04:51:41.720881 17396 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:41.721554 17396 heartbeater.cc:511] Master 127.13.148.190:44369 requested a full tablet report, sending...
I20260806 04:51:41.722865 17062 ts_manager.cc:194] Registered new tserver with Master: 9cb0463c12ae44fe8a4345d7534a302c (127.13.148.130:38221)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
I20260806 04:51:41.723821 17062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:44343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
I20260806 04:51:41.743649 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44369
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:42.039665 17401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:42.039918 17401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:42.040017 17401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:42.046382 17401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:42.046464 17401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:42.046552 17401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:51:42.054854 17401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:42.055859 17401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:42.057139 17401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:42.068928 17409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:42.069053 17406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:42.069023 17407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:42.069949 17401 server_base.cc:1034] running on GCE node
I20260806 04:51:42.070501 17401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:42.071262 17401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:42.072443 17401 hybrid_clock.cc:648] HybridClock initialized: now 1785991902072406 us; error 32 us; skew 500 ppm
I20260806 04:51:42.072708 17401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:42 dist-test-slave-1x4g krb5kdc[17015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991902, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:42.075515 17401 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:51:42.076941 17401 webserver.cc:460] Webserver started at http://127.13.148.131:40935/ using document root <none> and password file <none>
I20260806 04:51:42.077584 17401 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:42.077723 17401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:42.078024 17401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:42.081058 17401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "04aabd03484a43d5976540c0833dc484"
format_stamp: "Formatted at 2026-08-06 04:51:42 on dist-test-slave-1x4g"
server_key: "4901f43927ebe722756465a1aaf13b31"
server_key_iv: "6c43b5f59b974e6ac32307b319f5dc88"
server_key_version: "encryptionkey@0"
I20260806 04:51:42.081758 17401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "04aabd03484a43d5976540c0833dc484"
format_stamp: "Formatted at 2026-08-06 04:51:42 on dist-test-slave-1x4g"
server_key: "4901f43927ebe722756465a1aaf13b31"
server_key_iv: "6c43b5f59b974e6ac32307b319f5dc88"
server_key_version: "encryptionkey@0"
I20260806 04:51:42.086228 17401 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:51:42.089605 17416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:42.090781 17401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260806 04:51:42.090957 17401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "04aabd03484a43d5976540c0833dc484"
format_stamp: "Formatted at 2026-08-06 04:51:42 on dist-test-slave-1x4g"
server_key: "4901f43927ebe722756465a1aaf13b31"
server_key_iv: "6c43b5f59b974e6ac32307b319f5dc88"
server_key_version: "encryptionkey@0"
I20260806 04:51:42.091130 17401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:42.105749 17401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:42.109226 17401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:42.109551 17401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:42.110427 17401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:42.111609 17401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:42.111701 17401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:42.111833 17401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:42.111899 17401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:42.146015 17401 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:44507
I20260806 04:51:42.146065 17531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:44507 every 8 connection(s)
I20260806 04:51:42.147198 17401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:51:42.148916 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 17401
I20260806 04:51:42.149124 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:51:42.149680 13906 external_mini_cluster.cc:1428] Setting key 632bde130dc1cd085f4e4f8b80db111b
Aug 06 04:51:42 dist-test-slave-1x4g krb5kdc[17015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991902, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:42.165802 17259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:42.149787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:32821 (local address 127.13.148.190:44369)
0806 04:51:42.149986 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:42.149994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:42.151199 (+ 1205us) server_negotiation.cc:407] Connection header received
0806 04:51:42.152872 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:42.152876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:42.152932 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:42.153007 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:42.154297 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:42.154955 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:42.156613 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:42.157116 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:42.157240 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:42.160495 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:42.160501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:42.160510 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:42.160539 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:42.163112 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:42.163839 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:42.163843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:42.163847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:42.163994 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:42.164494 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:42.164499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:42.164502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:42.164767 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:42.164933 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:42.165450 (+ 517us) server_negotiation.cc:299] Negotiation successful
0806 04:51:42.165592 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 04:51:42.166699 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:42.150251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:44369 (local address 127.13.148.131:32821)
0806 04:51:42.151054 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:42.151096 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:42.152620 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:42.153191 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:42.153198 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:42.153623 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:42.154148 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:42.154164 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:42.155101 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:42.155106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:42.156469 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:42.156482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:42.157342 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:42.157350 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:42.157603 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:42.158186 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:42.158205 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:42.160291 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:42.163237 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:42.163243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:42.163253 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:42.163730 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:42.164121 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:42.164127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:42.164132 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:42.164322 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:42.164915 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:42.164920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:42.165180 (+ 260us) client_negotiation.cc:769] Sending connection context
0806 04:51:42.165380 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 04:51:42.165708 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":356,"threads_started":1}
I20260806 04:51:42.167857 17532 heartbeater.cc:347] Connected to a master server at 127.13.148.190:44369
I20260806 04:51:42.168133 17532 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:42.168877 17532 heartbeater.cc:511] Master 127.13.148.190:44369 requested a full tablet report, sending...
I20260806 04:51:42.170182 17062 ts_manager.cc:194] Registered new tserver with Master: 04aabd03484a43d5976540c0833dc484 (127.13.148.131:44507)
I20260806 04:51:42.170876 17062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:32821
I20260806 04:51:42.176499 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 04:51:42 dist-test-slave-1x4g krb5kdc[17015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991902, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 04:51:42 dist-test-slave-1x4g krb5kdc[17015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991902, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:42.220606 17259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:42.208069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35946 (local address 127.13.148.190:44369)
0806 04:51:42.208225 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:42.208232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:42.208677 (+ 445us) server_negotiation.cc:407] Connection header received
0806 04:51:42.208877 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:42.208881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:42.208942 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:42.209031 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:42.210391 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:42.211095 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:42.212206 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:42.212564 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:42.212648 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:42.216012 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:42.216016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:42.216019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:42.216046 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:42.218433 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:42.218999 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:42.219003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:42.219005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:42.219126 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:42.219485 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:42.219487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:42.219489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:42.219796 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:42.219959 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:42.220283 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 04:51:42.220416 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 04:51:42.221102 17062 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35946
I20260806 04:51:42.222294 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 17125
Aug 06 04:51:42 dist-test-slave-1x4g krb5kdc[17015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991902, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:51:42.235101 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:42.223110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.13.148.129:35487)
0806 04:51:42.223570 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:42.223577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:42.223600 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:51:42.223654 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:42.223660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:42.223835 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:42.223949 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:42.224884 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:42.225697 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:42.226690 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:42.227143 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:42.227225 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:42.230635 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:42.230644 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:42.230655 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:42.230682 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:42.233108 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:42.233596 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:42.233601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:42.233603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:42.233710 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:42.233972 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:42.233975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:42.233976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:42.234244 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:42.234439 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:42.234603 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 04:51:42.234797 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":241,"threads_started":1}
W20260806 04:51:42.236138 17252 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58212
W20260806 04:51:42.237102 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 17125: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:51:42.237265 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 17125
I20260806 04:51:42.254370 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 17263
Aug 06 04:51:42 dist-test-slave-1x4g krb5kdc[17015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991902, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:51:42.268509 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:42.255030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.13.148.130:38221)
0806 04:51:42.255465 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:42.255471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:42.255491 (+ 20us) server_negotiation.cc:407] Connection header received
0806 04:51:42.255540 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:42.255544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:42.255710 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:42.255866 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:42.257163 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:42.258130 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:42.259287 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:42.259828 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:42.259962 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:42.263822 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:42.263831 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:42.263841 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:42.263873 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:42.266164 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:42.266738 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:42.266743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:42.266745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:42.266856 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:42.267180 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:42.267184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:42.267187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:42.267467 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:42.267738 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:42.267991 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 04:51:42.268220 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":205,"threads_started":1}
W20260806 04:51:42.269796 17391 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42174
W20260806 04:51:42.270763 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 17263: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:51:42.270905 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 17263
I20260806 04:51:42.286381 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 17401
Aug 06 04:51:42 dist-test-slave-1x4g krb5kdc[17015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991902, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
I20260806 04:51:42.300077 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:42.287083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.13.148.131:44507)
0806 04:51:42.287561 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:42.287567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:42.287591 (+ 24us) server_negotiation.cc:407] Connection header received
0806 04:51:42.287638 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:42.287643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:42.287804 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:42.287904 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:42.289118 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:42.290095 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:42.291059 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:42.291513 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:42.291598 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:42.295320 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:42.295329 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:42.295340 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:42.295366 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:42.297943 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:42.298492 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:42.298501 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:42.298503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:42.298628 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:42.298919 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:42.298922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:42.298924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:42.299227 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:42.299451 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:42.299633 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 04:51:42.299827 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":219,"threads_started":1}
W20260806 04:51:42.301388 17526 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57712
W20260806 04:51:42.302345 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 17401: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:51:42.302481 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 17401
I20260806 04:51:42.319092 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 17030
I20260806 04:51:42.320179 17093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:42.414803 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 17030
2026-08-06T04:51:42Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4791 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:51:42 dist-test-slave-1x4g krb5kdc[17550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:51:42 dist-test-slave-1x4g krb5kdc[17550](info): set up 2 sockets
Aug 06 04:51:42 dist-test-slave-1x4g krb5kdc[17550](info): commencing operation
krb5kdc: starting...
W20260806 04:51:43.422084 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:51:43 dist-test-slave-1x4g krb5kdc[17550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:51:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:51:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:51:43.599967 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:42853
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:40417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:42853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:43.896764 17563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:43.897120 17563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:43.897274 17563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:43.903527 17563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:51:43.903643 17563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:43.903743 17563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:43.903821 17563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:51:43.903889 17563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:51:43.912462 17563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:40417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:42853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:42853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:43.913666 17563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:43.914893 17563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:43.926172 17571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:43.926206 17569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:43.926256 17568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:43.927124 17563 server_base.cc:1034] running on GCE node
I20260806 04:51:43.927814 17563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:43.929188 17563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:43.930423 17563 hybrid_clock.cc:648] HybridClock initialized: now 1785991903930385 us; error 50 us; skew 500 ppm
I20260806 04:51:43.930752 17563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:43 dist-test-slave-1x4g krb5kdc[17550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991903, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:43.933617 17563 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:51:43.934993 17563 webserver.cc:460] Webserver started at http://127.13.148.190:43985/ using document root <none> and password file <none>
I20260806 04:51:43.935611 17563 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:43.935742 17563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:43.936038 17563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:43.938987 17563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "d656d8c4a2894ff0b88fb615f5435ad0"
format_stamp: "Formatted at 2026-08-06 04:51:43 on dist-test-slave-1x4g"
server_key: "856638f56967e5a980c52be72a061128"
server_key_iv: "b32d61e6491efcaff8cd620546077bac"
server_key_version: "encryptionkey@0"
I20260806 04:51:43.939759 17563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "d656d8c4a2894ff0b88fb615f5435ad0"
format_stamp: "Formatted at 2026-08-06 04:51:43 on dist-test-slave-1x4g"
server_key: "856638f56967e5a980c52be72a061128"
server_key_iv: "b32d61e6491efcaff8cd620546077bac"
server_key_version: "encryptionkey@0"
I20260806 04:51:43.944343 17563 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260806 04:51:43.947422 17578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:43.948589 17563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:51:43.948810 17563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "d656d8c4a2894ff0b88fb615f5435ad0"
format_stamp: "Formatted at 2026-08-06 04:51:43 on dist-test-slave-1x4g"
server_key: "856638f56967e5a980c52be72a061128"
server_key_iv: "b32d61e6491efcaff8cd620546077bac"
server_key_version: "encryptionkey@0"
I20260806 04:51:43.949021 17563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:43.980414 17563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:43.985087 17563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:43.985355 17563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:44.002825 17563 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:42853
I20260806 04:51:44.002851 17630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:42853 every 8 connection(s)
I20260806 04:51:44.004055 17563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:51:44.007519 17631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:44.008307 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 17563
I20260806 04:51:44.008513 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:51:44.009232 13906 external_mini_cluster.cc:1428] Setting key af4c12df434dcf83aaef01cd002c3b02
I20260806 04:51:44.021032 17631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0: Bootstrap starting.
Aug 06 04:51:44 dist-test-slave-1x4g krb5kdc[17550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:44.024317 17631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:44.025471 17631 log.cc:824] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:44.028218 17631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0: No bootstrap required, opened a new log
I20260806 04:51:44.030993 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:44.011062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.13.148.190:42853)
0806 04:51:44.011884 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:44.011895 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:44.011938 (+ 43us) server_negotiation.cc:407] Connection header received
0806 04:51:44.013611 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:44.013631 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:44.013967 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:44.014270 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:44.015893 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:44.017183 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:44.019030 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:44.019656 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:44.019786 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:44.023979 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:44.023997 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:44.024009 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:44.024038 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:44.027069 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:44.027708 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:44.027714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:44.027720 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:44.027884 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:44.028237 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:44.028242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:44.028246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:44.028771 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:44.029119 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:44.029576 (+ 457us) server_negotiation.cc:299] Negotiation successful
0806 04:51:44.029995 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":379,"threads_started":1}
I20260806 04:51:44.035905 17631 raft_consensus.cc:348] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d656d8c4a2894ff0b88fb615f5435ad0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42853 } }
I20260806 04:51:44.036259 17631 raft_consensus.cc:374] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:44.036371 17631 raft_consensus.cc:729] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d656d8c4a2894ff0b88fb615f5435ad0, State: Initialized, Role: FOLLOWER
I20260806 04:51:44.036999 17631 consensus_queue.cc:260] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d656d8c4a2894ff0b88fb615f5435ad0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42853 } }
I20260806 04:51:44.037246 17631 raft_consensus.cc:388] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:51:44.037400 17631 raft_consensus.cc:482] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:51:44.037564 17631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:44.039970 17631 raft_consensus.cc:504] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d656d8c4a2894ff0b88fb615f5435ad0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42853 } }
I20260806 04:51:44.040407 17631 leader_election.cc:302] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d656d8c4a2894ff0b88fb615f5435ad0; no voters:
I20260806 04:51:44.040983 17631 leader_election.cc:288] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:51:44.041095 17636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:44.041685 17636 raft_consensus.cc:686] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [term 1 LEADER]: Becoming Leader. State: Replica: d656d8c4a2894ff0b88fb615f5435ad0, State: Running, Role: LEADER
I20260806 04:51:44.042171 17636 consensus_queue.cc:237] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d656d8c4a2894ff0b88fb615f5435ad0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42853 } }
I20260806 04:51:44.042714 17631 sys_catalog.cc:564] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:51:44.045068 17637 sys_catalog.cc:455] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d656d8c4a2894ff0b88fb615f5435ad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d656d8c4a2894ff0b88fb615f5435ad0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42853 } } }
I20260806 04:51:44.045508 17637 sys_catalog.cc:458] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:44.045337 17639 sys_catalog.cc:455] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d656d8c4a2894ff0b88fb615f5435ad0. Latest consensus state: current_term: 1 leader_uuid: "d656d8c4a2894ff0b88fb615f5435ad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d656d8c4a2894ff0b88fb615f5435ad0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42853 } } }
I20260806 04:51:44.045621 17639 sys_catalog.cc:458] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:44.047709 17646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:51:44.051772 17646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:51:44.060246 17646 catalog_manager.cc:1347] Generated new cluster ID: 7c4836a83d3748b4ac4aeec7fe0ca5ba
I20260806 04:51:44.060348 17646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:51:44.073817 17646 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:51:44.074863 17646 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:51:44.089140 17646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d656d8c4a2894ff0b88fb615f5435ad0: Generated new TSK 0
I20260806 04:51:44.089916 17646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:51:44.133737 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42853
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:40417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:44.436431 17658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:44.436769 17658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:44.436971 17658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:44.443678 17658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:44.443806 17658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:44.443986 17658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:51:44.453683 17658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:40417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:44.454878 17658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:44.456072 17658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:44.468150 17663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:44.468106 17664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:44.468060 17667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:44.469064 17658 server_base.cc:1034] running on GCE node
I20260806 04:51:44.469573 17658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:44.470347 17658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:44.471575 17658 hybrid_clock.cc:648] HybridClock initialized: now 1785991904471528 us; error 36 us; skew 500 ppm
I20260806 04:51:44.471855 17658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:44 dist-test-slave-1x4g krb5kdc[17550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991904, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:44.474689 17658 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:51:44.475979 17658 webserver.cc:460] Webserver started at http://127.13.148.129:40131/ using document root <none> and password file <none>
I20260806 04:51:44.476583 17658 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:44.476748 17658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:44.477074 17658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:44.479707 17658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "491b7622dbf945aba675c3caffdec059"
format_stamp: "Formatted at 2026-08-06 04:51:44 on dist-test-slave-1x4g"
server_key: "228f47a320f91962e7b6a011ceb0553e"
server_key_iv: "fdd9dd78d1f020b2ebece2e24e026e56"
server_key_version: "encryptionkey@0"
I20260806 04:51:44.480459 17658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "491b7622dbf945aba675c3caffdec059"
format_stamp: "Formatted at 2026-08-06 04:51:44 on dist-test-slave-1x4g"
server_key: "228f47a320f91962e7b6a011ceb0553e"
server_key_iv: "fdd9dd78d1f020b2ebece2e24e026e56"
server_key_version: "encryptionkey@0"
I20260806 04:51:44.484931 17658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260806 04:51:44.487923 17674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:44.489224 17658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:51:44.489398 17658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "491b7622dbf945aba675c3caffdec059"
format_stamp: "Formatted at 2026-08-06 04:51:44 on dist-test-slave-1x4g"
server_key: "228f47a320f91962e7b6a011ceb0553e"
server_key_iv: "fdd9dd78d1f020b2ebece2e24e026e56"
server_key_version: "encryptionkey@0"
I20260806 04:51:44.489594 17658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:44.509215 17658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:44.513437 17658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:44.513960 17658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:44.514863 17658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:44.516024 17658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:44.516124 17658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:44.516260 17658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:44.516332 17658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:44.547739 17658 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:32845
I20260806 04:51:44.547755 17790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:32845 every 8 connection(s)
I20260806 04:51:44.548980 17658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:51:44.550071 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 17658
I20260806 04:51:44.550258 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:51:44.551079 13906 external_mini_cluster.cc:1428] Setting key 08a56d890ad33348cd9c8a3be49a7f14
Aug 06 04:51:44 dist-test-slave-1x4g krb5kdc[17550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991904, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:44.570024 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:44.551616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:42937 (local address 127.13.148.190:42853)
0806 04:51:44.552088 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:44.552095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:44.553038 (+ 943us) server_negotiation.cc:407] Connection header received
0806 04:51:44.554768 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:44.554773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:44.554867 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:44.554995 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:44.556292 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:44.557066 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:44.558569 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:44.559016 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:44.559147 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:44.563086 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:44.563095 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:44.563103 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:44.563147 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:44.566692 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:44.567435 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:44.567460 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:44.567466 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:44.567623 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:44.568015 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:44.568022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:44.568028 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:44.568406 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:44.568571 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:44.569274 (+ 703us) server_negotiation.cc:299] Negotiation successful
0806 04:51:44.569530 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":240,"threads_started":1}
I20260806 04:51:44.570629 17794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:44.552118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42853 (local address 127.13.148.129:42937)
0806 04:51:44.552883 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:44.552939 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:44.554530 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:44.555214 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:44.555221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:44.555669 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:44.556140 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:44.556160 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:44.557238 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:44.557242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:44.558417 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:44.558429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:44.559243 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:44.559254 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:44.559604 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:44.560334 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:44.560367 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:44.562843 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:44.566891 (+ 4048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:44.566897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:44.566908 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:44.567298 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:44.567730 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:44.567733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:44.567735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:44.567905 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:44.568533 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:44.568538 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:44.568921 (+ 383us) client_negotiation.cc:769] Sending connection context
0806 04:51:44.569145 (+ 224us) client_negotiation.cc:240] Negotiation successful
0806 04:51:44.569514 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":269,"threads_started":1}
I20260806 04:51:44.572119 17791 heartbeater.cc:347] Connected to a master server at 127.13.148.190:42853
I20260806 04:51:44.572450 17791 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
I20260806 04:51:44.573383 17791 heartbeater.cc:511] Master 127.13.148.190:42853 requested a full tablet report, sending...
I20260806 04:51:44.575547 17596 ts_manager.cc:194] Registered new tserver with Master: 491b7622dbf945aba675c3caffdec059 (127.13.148.129:32845)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
I20260806 04:51:44.577167 17596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:42937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:51:44.598043 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42853
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:40417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:44.893941 17797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:44.894192 17797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:44.894351 17797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:44.900967 17797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:44.901055 17797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:44.901242 17797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:51:44.909361 17797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:40417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:44.910534 17797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:44.911736 17797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:44.924072 17803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:44.924114 17804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:44.924540 17806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:44.925180 17797 server_base.cc:1034] running on GCE node
I20260806 04:51:44.925776 17797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:44.926592 17797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:44.927786 17797 hybrid_clock.cc:648] HybridClock initialized: now 1785991904927733 us; error 25 us; skew 500 ppm
I20260806 04:51:44.928071 17797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:44 dist-test-slave-1x4g krb5kdc[17550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991904, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:44.931113 17797 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:51:44.932379 17797 webserver.cc:460] Webserver started at http://127.13.148.130:37993/ using document root <none> and password file <none>
I20260806 04:51:44.933017 17797 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:44.933108 17797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:44.933334 17797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:44.935972 17797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "6fce470eddb14f55aed2c3a0ebc64807"
format_stamp: "Formatted at 2026-08-06 04:51:44 on dist-test-slave-1x4g"
server_key: "bfe1600c9b11abb26de4423c3f43f89f"
server_key_iv: "591f8da9c4fdb80b9100ba97d971e8bd"
server_key_version: "encryptionkey@0"
I20260806 04:51:44.936720 17797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "6fce470eddb14f55aed2c3a0ebc64807"
format_stamp: "Formatted at 2026-08-06 04:51:44 on dist-test-slave-1x4g"
server_key: "bfe1600c9b11abb26de4423c3f43f89f"
server_key_iv: "591f8da9c4fdb80b9100ba97d971e8bd"
server_key_version: "encryptionkey@0"
I20260806 04:51:44.941579 17797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:51:44.944568 17813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:44.945768 17797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:51:44.945878 17797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "6fce470eddb14f55aed2c3a0ebc64807"
format_stamp: "Formatted at 2026-08-06 04:51:44 on dist-test-slave-1x4g"
server_key: "bfe1600c9b11abb26de4423c3f43f89f"
server_key_iv: "591f8da9c4fdb80b9100ba97d971e8bd"
server_key_version: "encryptionkey@0"
I20260806 04:51:44.946022 17797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:44.965767 17797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:44.969797 17797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:44.970096 17797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:44.970912 17797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:44.972127 17797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:44.972196 17797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:44.972365 17797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:44.972410 17797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:45.002461 17797 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:46807
I20260806 04:51:45.002552 17928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:46807 every 8 connection(s)
I20260806 04:51:45.003899 17797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:51:45.005448 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 17797
I20260806 04:51:45.005589 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:51:45.006093 13906 external_mini_cluster.cc:1428] Setting key 95cb4a26b13b819847ce68161569d2b5
Aug 06 04:51:45 dist-test-slave-1x4g krb5kdc[17550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991904, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
I20260806 04:51:45.026330 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:45.006314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:37357 (local address 127.13.148.190:42853)
0806 04:51:45.006531 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:45.006537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:45.007756 (+ 1219us) server_negotiation.cc:407] Connection header received
0806 04:51:45.010308 (+ 2552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:45.010314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:45.010403 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:45.010464 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:45.012409 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.013415 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.015487 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.016181 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.016308 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:45.020640 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:45.020646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:45.020650 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:45.020683 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:45.023525 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:45.024352 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:45.024355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:45.024357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:45.024499 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:45.024954 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:45.024957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:45.024958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:45.025227 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:45.025339 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:45.025915 (+ 576us) server_negotiation.cc:299] Negotiation successful
0806 04:51:45.026089 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 04:51:45.027269 17931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:45.006852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42853 (local address 127.13.148.130:37357)
0806 04:51:45.007613 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:45.007666 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:45.010031 (+ 2365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:45.010905 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:45.010914 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:45.011505 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:45.012236 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:45.012258 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.013616 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.013621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:45.015334 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:45.015347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.016386 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.016391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:45.016659 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:45.017430 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:45.017461 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:45.020430 (+ 2969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:45.023659 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:45.023667 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:45.023683 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:45.024250 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:45.024620 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:45.024624 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:45.024628 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:45.024853 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:45.025349 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:45.025355 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:45.025624 (+ 269us) client_negotiation.cc:769] Sending connection context
0806 04:51:45.025821 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 04:51:45.026212 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":292,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
I20260806 04:51:45.028589 17929 heartbeater.cc:347] Connected to a master server at 127.13.148.190:42853
I20260806 04:51:45.028918 17929 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:45.029597 17929 heartbeater.cc:511] Master 127.13.148.190:42853 requested a full tablet report, sending...
I20260806 04:51:45.031006 17596 ts_manager.cc:194] Registered new tserver with Master: 6fce470eddb14f55aed2c3a0ebc64807 (127.13.148.130:46807)
I20260806 04:51:45.031756 17596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:37357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
I20260806 04:51:45.049431 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42853
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:40417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:45.352028 17934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:45.352272 17934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:45.352357 17934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:45.358860 17934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:45.358951 17934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:45.359157 17934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:51:45.367604 17934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:40417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:45.368702 17934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:45.370000 17934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:45.381948 17939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:45.381953 17942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:45.381973 17940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:45.382788 17934 server_base.cc:1034] running on GCE node
I20260806 04:51:45.383358 17934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:45.384233 17934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:45.385456 17934 hybrid_clock.cc:648] HybridClock initialized: now 1785991905385422 us; error 56 us; skew 500 ppm
I20260806 04:51:45.385740 17934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:45 dist-test-slave-1x4g krb5kdc[17550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991905, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:45.388686 17934 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:51:45.390025 17934 webserver.cc:460] Webserver started at http://127.13.148.131:39255/ using document root <none> and password file <none>
I20260806 04:51:45.390559 17934 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:45.390646 17934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:45.390870 17934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:45.393560 17934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "b801face85054eacaee25c724c104b6a"
format_stamp: "Formatted at 2026-08-06 04:51:45 on dist-test-slave-1x4g"
server_key: "e7ec6079f4a4bf68d3f0587519e2c2e1"
server_key_iv: "a94740033fc5d8e3e6131087a54940ba"
server_key_version: "encryptionkey@0"
I20260806 04:51:45.394331 17934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "b801face85054eacaee25c724c104b6a"
format_stamp: "Formatted at 2026-08-06 04:51:45 on dist-test-slave-1x4g"
server_key: "e7ec6079f4a4bf68d3f0587519e2c2e1"
server_key_iv: "a94740033fc5d8e3e6131087a54940ba"
server_key_version: "encryptionkey@0"
I20260806 04:51:45.398979 17934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 04:51:45.402029 17949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:45.403148 17934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260806 04:51:45.403266 17934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "b801face85054eacaee25c724c104b6a"
format_stamp: "Formatted at 2026-08-06 04:51:45 on dist-test-slave-1x4g"
server_key: "e7ec6079f4a4bf68d3f0587519e2c2e1"
server_key_iv: "a94740033fc5d8e3e6131087a54940ba"
server_key_version: "encryptionkey@0"
I20260806 04:51:45.403415 17934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:45.433231 17934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:45.437101 17934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:45.437332 17934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:45.438158 17934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:45.439391 17934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:45.439462 17934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:45.439553 17934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:45.439599 17934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:45.470352 17934 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:41199
I20260806 04:51:45.470423 18063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:41199 every 8 connection(s)
I20260806 04:51:45.471458 17934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:51:45.475407 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 17934
I20260806 04:51:45.475589 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:51:45.476081 13906 external_mini_cluster.cc:1428] Setting key cdc64a53de8e9542f9da725f33c8e8cb
Aug 06 04:51:45 dist-test-slave-1x4g krb5kdc[17550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991905, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:45.488701 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:45.473871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:36605 (local address 127.13.148.190:42853)
0806 04:51:45.474035 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:45.474039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:45.475039 (+ 1000us) server_negotiation.cc:407] Connection header received
0806 04:51:45.477265 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:45.477270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:45.477369 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:45.477477 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:45.479162 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.479959 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.481317 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.481662 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.481744 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:45.484429 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:45.484434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:45.484437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:45.484463 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:45.486600 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:45.487129 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:45.487132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:45.487135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:45.487229 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:45.487533 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:45.487535 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:45.487537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:45.487745 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:45.487845 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:45.488327 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 04:51:45.488482 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:51:45.489629 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:45.474281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42853 (local address 127.13.148.131:36605)
0806 04:51:45.474865 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:45.474897 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:45.477006 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:45.477690 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:45.477700 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:45.478397 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:45.478964 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:45.478981 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.480093 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.480096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:45.481208 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:45.481219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.481822 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.481826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:45.481994 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:45.482489 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:45.482506 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:45.484272 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:45.486699 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:45.486704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:45.486712 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:45.487045 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:45.487308 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:45.487310 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:45.487312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:45.487449 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:45.487834 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:45.487838 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:45.488084 (+ 246us) client_negotiation.cc:769] Sending connection context
0806 04:51:45.488267 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 04:51:45.488574 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":250,"threads_started":1}
I20260806 04:51:45.491003 18064 heartbeater.cc:347] Connected to a master server at 127.13.148.190:42853
I20260806 04:51:45.491326 18064 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:45.492074 18064 heartbeater.cc:511] Master 127.13.148.190:42853 requested a full tablet report, sending...
I20260806 04:51:45.493392 17596 ts_manager.cc:194] Registered new tserver with Master: b801face85054eacaee25c724c104b6a (127.13.148.131:41199)
I20260806 04:51:45.494165 17596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:36605
I20260806 04:51:45.503078 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 04:51:45 dist-test-slave-1x4g krb5kdc[17550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 04:51:45 dist-test-slave-1x4g krb5kdc[17550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:51:45.549301 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:45.534551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49042 (local address 127.13.148.129:32845)
0806 04:51:45.535067 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:45.535075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:45.535102 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:51:45.535255 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:45.535261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:45.535471 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:45.535618 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:45.537001 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.538273 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.539527 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.540080 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.540211 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:45.543869 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:45.543878 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:45.543891 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:45.543932 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:45.546491 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:45.547143 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:45.547151 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:45.547156 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:45.547308 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:45.547628 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:45.547632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:45.547635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:45.548054 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:45.548371 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:45.548547 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 04:51:45.548880 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":244,"threads_started":1}
Aug 06 04:51:45 dist-test-slave-1x4g krb5kdc[17550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:45.565129 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:45.553440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40506 (local address 127.13.148.190:42853)
0806 04:51:45.553619 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:45.553625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:45.553802 (+ 177us) server_negotiation.cc:407] Connection header received
0806 04:51:45.553929 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:45.553932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:45.553997 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:45.554069 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:45.555069 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.555692 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.556713 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.557158 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.557254 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:45.560686 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:45.560690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:45.560693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:45.560739 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:45.562967 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:45.563606 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:45.563610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:45.563612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:45.563726 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:45.564031 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:45.564034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:45.564036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:45.564301 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:45.564514 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:45.564725 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 04:51:45.564920 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 04:51:45.565598 17596 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40506
I20260806 04:51:45.566502 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 17658
I20260806 04:51:45.578150 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:45.567249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49050 (local address 127.13.148.129:32845)
0806 04:51:45.567453 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:45.567458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:45.567657 (+ 199us) server_negotiation.cc:407] Connection header received
0806 04:51:45.567832 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:45.567837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:45.567935 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:45.568026 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:45.569165 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.569820 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.570969 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.571316 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.571445 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:45.573532 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:45.573537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:45.573539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:45.573566 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:45.575820 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:45.576563 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:45.576566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:45.576568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:45.576708 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:45.577118 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:45.577121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:45.577123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:45.577357 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:45.577521 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:45.577780 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 04:51:45.577937 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 04:51:45.578739 17786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:45.654986 17791 heartbeater.cc:503] Master 127.13.148.190:42853 was elected leader, sending a full tablet report...
I20260806 04:51:45.655515 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 17658
W20260806 04:51:45.672655 17580 connection.cc:535] server connection from 127.13.148.129:42937 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 04:51:45.672957 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 17797
Aug 06 04:51:45 dist-test-slave-1x4g krb5kdc[17550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:51:45.691859 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:45.673764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33910 (local address 127.13.148.130:46807)
0806 04:51:45.674271 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:45.674277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:45.674301 (+ 24us) server_negotiation.cc:407] Connection header received
0806 04:51:45.674360 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:45.674366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:45.674559 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:45.674687 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:45.675726 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.676648 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.678076 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.678601 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.678692 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:45.686795 (+ 8103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:45.686806 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:45.686819 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:45.686846 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:45.689510 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:45.690140 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:45.690146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:45.690148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:45.690253 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:45.690659 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:45.690661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:45.690663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:45.690944 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:45.691141 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:45.691357 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 04:51:45.691549 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":266,"threads_started":1}
I20260806 04:51:45.693042 17924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:45.767765 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 17797
I20260806 04:51:45.784349 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 17934
Aug 06 04:51:45 dist-test-slave-1x4g krb5kdc[17550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
I20260806 04:51:45.800053 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:45.785278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59070 (local address 127.13.148.131:41199)
0806 04:51:45.785852 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:45.785859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:45.785899 (+ 40us) server_negotiation.cc:407] Connection header received
0806 04:51:45.785956 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:45.785961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:45.786155 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:45.786318 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:45.787590 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.788588 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.790020 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:45.790570 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:45.790696 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:45.794577 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:45.794587 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:45.794601 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:45.794641 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:45.797332 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:45.797990 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:45.797997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:45.797999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:45.798111 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:45.798460 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:45.798465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:45.798468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:45.798787 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:45.799134 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:45.799427 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 04:51:45.799693 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":262,"threads_started":1}
I20260806 04:51:45.801280 18059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:45.886830 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 17934
I20260806 04:51:45.903363 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 17563
I20260806 04:51:45.904453 17626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:45.999691 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 17563
2026-08-06T04:51:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3584 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:51:46 dist-test-slave-1x4g krb5kdc[18084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:51:46 dist-test-slave-1x4g krb5kdc[18084](info): set up 2 sockets
Aug 06 04:51:46 dist-test-slave-1x4g krb5kdc[18084](info): commencing operation
krb5kdc: starting...
W20260806 04:51:47.022042 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:51:47 dist-test-slave-1x4g krb5kdc[18084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:51:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:51:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:51:47.196606 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:36751
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:36751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:47.493062 18097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:47.493323 18097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:47.493470 18097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:47.499758 18097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:51:47.499847 18097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:47.499940 18097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:47.500048 18097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:51:47.500157 18097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:51:47.508649 18097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:36751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:36751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:47.509848 18097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:47.511071 18097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:47.521764 18102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:47.521816 18105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:47.521716 18103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:47.523115 18097 server_base.cc:1034] running on GCE node
I20260806 04:51:47.523752 18097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:47.524955 18097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:47.526207 18097 hybrid_clock.cc:648] HybridClock initialized: now 1785991907526173 us; error 42 us; skew 500 ppm
I20260806 04:51:47.526510 18097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:47 dist-test-slave-1x4g krb5kdc[18084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991907, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:47.529196 18097 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:51:47.530256 18097 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:47.530375 18097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:47.530737 18097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:47.533479 18097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "08f0251a429343c397e5a8025984209a"
format_stamp: "Formatted at 2026-08-06 04:51:47 on dist-test-slave-1x4g"
server_key: "0a4511e303a00585395a65ea2860afae"
server_key_iv: "75502ca1506977342e1b8a293b8d3282"
server_key_version: "encryptionkey@0"
I20260806 04:51:47.534157 18097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "08f0251a429343c397e5a8025984209a"
format_stamp: "Formatted at 2026-08-06 04:51:47 on dist-test-slave-1x4g"
server_key: "0a4511e303a00585395a65ea2860afae"
server_key_iv: "75502ca1506977342e1b8a293b8d3282"
server_key_version: "encryptionkey@0"
I20260806 04:51:47.538697 18097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:51:47.541826 18111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:47.542968 18097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:51:47.543146 18097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "08f0251a429343c397e5a8025984209a"
format_stamp: "Formatted at 2026-08-06 04:51:47 on dist-test-slave-1x4g"
server_key: "0a4511e303a00585395a65ea2860afae"
server_key_iv: "75502ca1506977342e1b8a293b8d3282"
server_key_version: "encryptionkey@0"
I20260806 04:51:47.543380 18097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:47.553411 18097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:47.556896 18097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:47.557152 18097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:47.574011 18097 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:36751
I20260806 04:51:47.574030 18164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:36751 every 8 connection(s)
I20260806 04:51:47.574797 18097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:51:47.578130 18166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:47.582628 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 18097
I20260806 04:51:47.582813 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:51:47.583351 13906 external_mini_cluster.cc:1428] Setting key 206f3bc9298a2faf13704fc0024a8584
I20260806 04:51:47.587740 18166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a: Bootstrap starting.
I20260806 04:51:47.591325 18166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:47.592242 18166 log.cc:824] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:47.594715 18166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a: No bootstrap required, opened a new log
Aug 06 04:51:47 dist-test-slave-1x4g krb5kdc[18084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:47.603099 18166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f0251a429343c397e5a8025984209a" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36751 } }
I20260806 04:51:47.603541 18166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:47.603645 18166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08f0251a429343c397e5a8025984209a, State: Initialized, Role: FOLLOWER
I20260806 04:51:47.604156 18166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f0251a429343c397e5a8025984209a" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36751 } }
I20260806 04:51:47.604372 18166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:51:47.604504 18166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:51:47.604538 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:47.585140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.13.148.190:36751)
0806 04:51:47.585903 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:47.585916 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:47.585959 (+ 43us) server_negotiation.cc:407] Connection header received
0806 04:51:47.587476 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:47.587495 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:47.587817 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:47.588161 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:47.590080 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:47.591321 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:47.592544 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:47.593600 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:47.593776 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:47.597773 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:47.597782 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:47.597795 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:47.597831 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:47.600827 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:47.601542 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:47.601547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:47.601552 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:47.601700 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:47.602042 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:47.602044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:47.602046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:47.602474 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:47.602676 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:47.603069 (+ 393us) server_negotiation.cc:299] Negotiation successful
0806 04:51:47.603414 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":386,"threads_started":1}
I20260806 04:51:47.604702 18166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:47.607247 18166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f0251a429343c397e5a8025984209a" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36751 } }
I20260806 04:51:47.607699 18166 leader_election.cc:302] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08f0251a429343c397e5a8025984209a; no voters:
I20260806 04:51:47.608202 18166 leader_election.cc:288] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:51:47.608291 18171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:47.609112 18171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [term 1 LEADER]: Becoming Leader. State: Replica: 08f0251a429343c397e5a8025984209a, State: Running, Role: LEADER
I20260806 04:51:47.609592 18171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f0251a429343c397e5a8025984209a" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36751 } }
I20260806 04:51:47.610319 18166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:51:47.612023 18172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08f0251a429343c397e5a8025984209a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f0251a429343c397e5a8025984209a" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36751 } } }
I20260806 04:51:47.612345 18172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:47.612840 18173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08f0251a429343c397e5a8025984209a. Latest consensus state: current_term: 1 leader_uuid: "08f0251a429343c397e5a8025984209a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f0251a429343c397e5a8025984209a" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36751 } } }
I20260806 04:51:47.613265 18173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:47.615383 18176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:51:47.621094 18176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:51:47.628698 18176 catalog_manager.cc:1347] Generated new cluster ID: 1ddee2d468b240b2a3d959307149fda3
I20260806 04:51:47.628820 18176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:51:47.656742 18176 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:51:47.658530 18176 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:51:47.665342 18176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a: Generated new TSK 0
I20260806 04:51:47.666281 18176 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:51:47.710609 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:36751
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:48.013525 18192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:48.013743 18192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:48.013891 18192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:48.020764 18192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:48.020924 18192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:48.021100 18192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:51:48.029547 18192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:36751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:48.030691 18192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:48.031860 18192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:48.043833 18198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:48.043829 18200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:48.043896 18197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:48.044534 18192 server_base.cc:1034] running on GCE node
I20260806 04:51:48.045107 18192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:48.045925 18192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:48.047122 18192 hybrid_clock.cc:648] HybridClock initialized: now 1785991908047080 us; error 37 us; skew 500 ppm
I20260806 04:51:48.047431 18192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:48 dist-test-slave-1x4g krb5kdc[18084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991908, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:48.050442 18192 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:51:48.051525 18192 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:48.051631 18192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:48.051941 18192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:48.054442 18192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "4147287d60c94b3b8ff64fa61f898bf6"
format_stamp: "Formatted at 2026-08-06 04:51:48 on dist-test-slave-1x4g"
server_key: "ea0dabdaf3241dcfd3147d4883a0c4b5"
server_key_iv: "ebcc50dcf6399a37806b642bc30e0458"
server_key_version: "encryptionkey@0"
I20260806 04:51:48.055150 18192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "4147287d60c94b3b8ff64fa61f898bf6"
format_stamp: "Formatted at 2026-08-06 04:51:48 on dist-test-slave-1x4g"
server_key: "ea0dabdaf3241dcfd3147d4883a0c4b5"
server_key_iv: "ebcc50dcf6399a37806b642bc30e0458"
server_key_version: "encryptionkey@0"
I20260806 04:51:48.059804 18192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:51:48.063056 18206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:48.064244 18192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:51:48.064410 18192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "4147287d60c94b3b8ff64fa61f898bf6"
format_stamp: "Formatted at 2026-08-06 04:51:48 on dist-test-slave-1x4g"
server_key: "ea0dabdaf3241dcfd3147d4883a0c4b5"
server_key_iv: "ebcc50dcf6399a37806b642bc30e0458"
server_key_version: "encryptionkey@0"
I20260806 04:51:48.064596 18192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:48.083628 18192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:48.087096 18192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:48.087338 18192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:48.088146 18192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:48.089331 18192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:48.089442 18192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:48.089577 18192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:48.089673 18192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:48.120549 18192 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:34191
I20260806 04:51:48.120622 18321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:34191 every 8 connection(s)
I20260806 04:51:48.121337 18192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:51:48.126839 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 18192
I20260806 04:51:48.127033 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:51:48.127822 13906 external_mini_cluster.cc:1428] Setting key c02781f0d90e37e5f93e5762a98aee9f
Aug 06 04:51:48 dist-test-slave-1x4g krb5kdc[18084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991908, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:48.140609 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:48.123546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:59645 (local address 127.13.148.190:36751)
0806 04:51:48.123895 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:48.123899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:48.124829 (+ 930us) server_negotiation.cc:407] Connection header received
0806 04:51:48.126590 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:48.126594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:48.126666 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:48.126759 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:48.128673 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:48.129737 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:48.131138 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:48.131525 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:48.131654 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:48.134751 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:48.134755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:48.134758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:48.134785 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:48.137504 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:48.138368 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:48.138372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:48.138375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:48.138508 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:48.138912 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:48.138915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:48.138916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:48.139193 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:48.139303 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:48.140138 (+ 835us) server_negotiation.cc:299] Negotiation successful
0806 04:51:48.140322 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":174,"threads_started":1}
I20260806 04:51:48.141988 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:48.124047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:36751 (local address 127.13.148.129:59645)
0806 04:51:48.124654 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:48.124695 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:48.126276 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:48.127027 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:48.127038 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:48.127741 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:48.128514 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:48.128533 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:48.129880 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:48.129890 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:48.131003 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:48.131013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:48.131743 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:48.131749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:48.131965 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:48.132578 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:48.132598 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:48.134557 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:48.137647 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:48.137658 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:48.137675 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:48.138251 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:48.138608 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:48.138615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:48.138620 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:48.138820 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:48.139326 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:48.139337 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:48.139750 (+ 413us) client_negotiation.cc:769] Sending connection context
0806 04:51:48.140053 (+ 303us) client_negotiation.cc:240] Negotiation successful
0806 04:51:48.140653 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":253,"threads_started":1}
I20260806 04:51:48.143338 18322 heartbeater.cc:347] Connected to a master server at 127.13.148.190:36751
I20260806 04:51:48.143636 18322 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:48.144409 18322 heartbeater.cc:511] Master 127.13.148.190:36751 requested a full tablet report, sending...
I20260806 04:51:48.146631 18128 ts_manager.cc:194] Registered new tserver with Master: 4147287d60c94b3b8ff64fa61f898bf6 (127.13.148.129:34191)
I20260806 04:51:48.148111 18128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:59645
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:51:48.173246 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:36751
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:48.470084 18328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:48.470343 18328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:48.470431 18328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:48.476703 18328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:48.476814 18328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:48.476920 18328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:51:48.485525 18328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:36751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:48.486579 18328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:48.487859 18328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:48.499787 18333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:48.499753 18336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:48.499850 18334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:48.501050 18328 server_base.cc:1034] running on GCE node
I20260806 04:51:48.501587 18328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:48.502388 18328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:48.503568 18328 hybrid_clock.cc:648] HybridClock initialized: now 1785991908503521 us; error 46 us; skew 500 ppm
I20260806 04:51:48.503839 18328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:48 dist-test-slave-1x4g krb5kdc[18084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991908, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:48.506631 18328 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:51:48.507606 18328 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:48.507721 18328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:48.507994 18328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:48.510871 18328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "ca996479cbfa4a0fa504964c5fe06fc6"
format_stamp: "Formatted at 2026-08-06 04:51:48 on dist-test-slave-1x4g"
server_key: "f69b59093d04c35a06c53a5020844cc9"
server_key_iv: "7a2b0a0646c8c0a5b38df816b0b02fd2"
server_key_version: "encryptionkey@0"
I20260806 04:51:48.511657 18328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "ca996479cbfa4a0fa504964c5fe06fc6"
format_stamp: "Formatted at 2026-08-06 04:51:48 on dist-test-slave-1x4g"
server_key: "f69b59093d04c35a06c53a5020844cc9"
server_key_iv: "7a2b0a0646c8c0a5b38df816b0b02fd2"
server_key_version: "encryptionkey@0"
I20260806 04:51:48.516136 18328 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260806 04:51:48.519374 18342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:48.520569 18328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:51:48.520727 18328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "ca996479cbfa4a0fa504964c5fe06fc6"
format_stamp: "Formatted at 2026-08-06 04:51:48 on dist-test-slave-1x4g"
server_key: "f69b59093d04c35a06c53a5020844cc9"
server_key_iv: "7a2b0a0646c8c0a5b38df816b0b02fd2"
server_key_version: "encryptionkey@0"
I20260806 04:51:48.520995 18328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:48.537241 18328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:48.540834 18328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:48.541090 18328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:48.541898 18328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:48.543008 18328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:48.543120 18328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:48.543248 18328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:48.543345 18328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:48.574647 18328 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:32829
I20260806 04:51:48.574695 18457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:32829 every 8 connection(s)
I20260806 04:51:48.575421 18328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:51:48.579620 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 18328
I20260806 04:51:48.579795 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:51:48.580365 13906 external_mini_cluster.cc:1428] Setting key dcb17323172ee9702cef107a0aae66e3
Aug 06 04:51:48 dist-test-slave-1x4g krb5kdc[18084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991908, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:48.597218 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:48.577893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:60751 (local address 127.13.148.190:36751)
0806 04:51:48.578088 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:48.578093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:48.579230 (+ 1137us) server_negotiation.cc:407] Connection header received
0806 04:51:48.581816 (+ 2586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:48.581821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:48.581910 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:48.582012 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:48.583866 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:48.584847 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:48.587005 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:48.587589 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:48.587710 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:48.591738 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:48.591742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:48.591745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:48.591776 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:48.594233 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:48.595082 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:48.595085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:48.595087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:48.595208 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:48.595641 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:48.595644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:48.595646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:48.595862 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:48.595957 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:48.596824 (+ 867us) server_negotiation.cc:299] Negotiation successful
0806 04:51:48.596989 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 04:51:48.598740 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:48.578329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:36751 (local address 127.13.148.130:60751)
0806 04:51:48.579044 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:48.579106 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:48.581558 (+ 2452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:48.582243 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:48.582255 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:48.582930 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:48.583676 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:48.583714 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:48.584992 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:48.584999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:48.586856 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:48.586873 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:48.587832 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:48.587841 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:48.588146 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:48.588946 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:48.588992 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:48.591525 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:48.594378 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:48.594388 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:48.594405 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:48.594976 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:48.595315 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:48.595322 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:48.595327 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:48.595545 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:48.595978 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:48.595987 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:48.596405 (+ 418us) client_negotiation.cc:769] Sending connection context
0806 04:51:48.596718 (+ 313us) client_negotiation.cc:240] Negotiation successful
0806 04:51:48.597297 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":262,"threads_started":1}
I20260806 04:51:48.600256 18458 heartbeater.cc:347] Connected to a master server at 127.13.148.190:36751
I20260806 04:51:48.600596 18458 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:48.601380 18458 heartbeater.cc:511] Master 127.13.148.190:36751 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
I20260806 04:51:48.602835 18128 ts_manager.cc:194] Registered new tserver with Master: ca996479cbfa4a0fa504964c5fe06fc6 (127.13.148.130:32829)
I20260806 04:51:48.603557 18128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:60751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
I20260806 04:51:48.625766 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:36751
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:48.917258 18464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:48.917505 18464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:48.917598 18464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:48.923986 18464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:48.924074 18464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:48.924234 18464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:51:48.933068 18464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:36751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:48.934096 18464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:48.935160 18464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:48.946853 18469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:48.946897 18470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:48.946805 18472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:48.947606 18464 server_base.cc:1034] running on GCE node
I20260806 04:51:48.948060 18464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:48.948756 18464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:48.950037 18464 hybrid_clock.cc:648] HybridClock initialized: now 1785991908950006 us; error 30 us; skew 500 ppm
I20260806 04:51:48.950356 18464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:48 dist-test-slave-1x4g krb5kdc[18084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991908, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:48.953063 18464 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:51:48.954067 18464 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:48.954185 18464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:48.954455 18464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:48.957024 18464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "4751b0386c4c4a96b2b3652cd1d8dca8"
format_stamp: "Formatted at 2026-08-06 04:51:48 on dist-test-slave-1x4g"
server_key: "d842d77521466a195414b53642a559b1"
server_key_iv: "372331c83c666f5be82c264c946ee824"
server_key_version: "encryptionkey@0"
I20260806 04:51:48.957808 18464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "4751b0386c4c4a96b2b3652cd1d8dca8"
format_stamp: "Formatted at 2026-08-06 04:51:48 on dist-test-slave-1x4g"
server_key: "d842d77521466a195414b53642a559b1"
server_key_iv: "372331c83c666f5be82c264c946ee824"
server_key_version: "encryptionkey@0"
I20260806 04:51:48.962246 18464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 04:51:48.965262 18479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:48.966303 18464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:51:48.966457 18464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "4751b0386c4c4a96b2b3652cd1d8dca8"
format_stamp: "Formatted at 2026-08-06 04:51:48 on dist-test-slave-1x4g"
server_key: "d842d77521466a195414b53642a559b1"
server_key_iv: "372331c83c666f5be82c264c946ee824"
server_key_version: "encryptionkey@0"
I20260806 04:51:48.966650 18464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:48.979476 18464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:48.983219 18464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:48.983458 18464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:48.984257 18464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:48.985435 18464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:48.985549 18464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:48.985656 18464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:48.985733 18464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:49.017390 18464 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:38093
I20260806 04:51:49.017477 18594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:38093 every 8 connection(s)
I20260806 04:51:49.018260 18464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:51:49.021930 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 18464
I20260806 04:51:49.022137 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:51:49.022676 13906 external_mini_cluster.cc:1428] Setting key f268fd5f0b6c40337e3e9f1c688f739b
Aug 06 04:51:49 dist-test-slave-1x4g krb5kdc[18084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991908, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:49.037050 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:49.020753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:36009 (local address 127.13.148.190:36751)
0806 04:51:49.021059 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:49.021064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:49.022377 (+ 1313us) server_negotiation.cc:407] Connection header received
0806 04:51:49.024390 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:49.024395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:49.024482 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:49.024590 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:49.026089 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.026765 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.028133 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.028508 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.028629 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:49.031793 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:49.031798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:49.031801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:49.031829 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:49.034422 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:49.035030 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:49.035033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:49.035035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:49.035142 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:49.035478 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:49.035481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:49.035482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:49.035705 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:49.035807 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:49.036463 (+ 656us) server_negotiation.cc:299] Negotiation successful
0806 04:51:49.036675 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260806 04:51:49.038173 18597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:49.021356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:36751 (local address 127.13.148.131:36009)
0806 04:51:49.022193 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:49.022240 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:49.024120 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:49.024878 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:49.024885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:49.025331 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:49.025921 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:49.025937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.026903 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.026912 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:49.028004 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:49.028015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.028708 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.028717 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:49.028969 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:49.029580 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:49.029605 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:49.031624 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:49.034546 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:49.034552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:49.034561 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:49.034933 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:49.035222 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:49.035225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:49.035227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:49.035379 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:49.035824 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:49.035832 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:49.036127 (+ 295us) client_negotiation.cc:769] Sending connection context
0806 04:51:49.036331 (+ 204us) client_negotiation.cc:240] Negotiation successful
0806 04:51:49.036902 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":326,"threads_started":1}
I20260806 04:51:49.039306 18595 heartbeater.cc:347] Connected to a master server at 127.13.148.190:36751
I20260806 04:51:49.039547 18595 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:49.040184 18595 heartbeater.cc:511] Master 127.13.148.190:36751 requested a full tablet report, sending...
I20260806 04:51:49.041465 18128 ts_manager.cc:194] Registered new tserver with Master: 4751b0386c4c4a96b2b3652cd1d8dca8 (127.13.148.131:38093)
I20260806 04:51:49.042097 18128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:36009
I20260806 04:51:49.049556 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:51:49.065162 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:49.053615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.13.148.190:36751)
0806 04:51:49.053752 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:49.053758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:49.054153 (+ 395us) server_negotiation.cc:407] Connection header received
0806 04:51:49.054323 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:49.054328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:49.054399 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:49.054489 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:49.055694 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.056273 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.057653 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.058108 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.058233 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:49.060444 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:49.060449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:49.060452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:49.060478 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:49.062716 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:49.063345 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:49.063350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:49.063353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:49.063475 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:49.063833 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:49.063838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:49.063841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:49.064127 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:49.064381 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:49.064671 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 04:51:49.064896 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:51:49.068586 18128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:51:49.075568 18128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:51:49.097245 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:49.087129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35150 (local address 127.13.148.130:32829)
0806 04:51:49.088003 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:49.088012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:49.088039 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:51:49.088143 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:49.088152 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:49.088404 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:49.089396 (+ 992us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:49.089689 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.091489 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.094986 (+ 3497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.095949 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.096139 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:49.096307 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:49.096653 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 04:51:49.096858 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":1663,"threads_started":1}
I20260806 04:51:49.097249 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:49.086488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:32829 (local address 127.0.0.1:35150)
0806 04:51:49.087252 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:49.087290 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:49.087443 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:49.088637 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:49.088648 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:49.088682 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:49.089137 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:49.089156 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.092233 (+ 3077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.092245 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:49.094839 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:49.094851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.096139 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.096145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:49.096345 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:49.096405 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 04:51:49.096609 (+ 204us) client_negotiation.cc:240] Negotiation successful
0806 04:51:49.096859 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":806,"threads_started":1}
I20260806 04:51:49.100920 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:49.086513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:34191 (local address 127.0.0.1:54660)
0806 04:51:49.092648 (+ 6135us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:49.092666 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:49.092791 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:49.093459 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:49.093463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:49.093484 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:49.093940 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:49.093952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.095896 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.095901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:49.098331 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:49.098345 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.100190 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.100197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:49.100360 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:49.100389 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 04:51:49.100481 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 04:51:49.100593 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6000,"thread_start_us":5848,"threads_started":1}
I20260806 04:51:49.101052 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:49.086539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.13.148.129:34191)
0806 04:51:49.092097 (+ 5558us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:49.092107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:49.092922 (+ 815us) server_negotiation.cc:407] Connection header received
0806 04:51:49.093044 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:49.093054 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:49.093276 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:49.093459 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:49.094105 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.095751 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.098522 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.099970 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.100154 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:49.100341 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:49.100538 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 04:51:49.100699 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5357,"thread_start_us":5211,"threads_started":1}
I20260806 04:51:49.101316 18611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:49.088017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:38093 (local address 127.0.0.1:54530)
0806 04:51:49.093422 (+ 5405us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:49.093438 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:49.093562 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:49.094354 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:49.094357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:49.094388 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:49.094728 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:49.094739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.096485 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.096490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:49.098551 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:49.098562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.100586 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.100592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:49.100745 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:49.100796 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 04:51:49.100890 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 04:51:49.101000 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5274,"thread_start_us":5134,"threads_started":1}
I20260806 04:51:49.101260 18392 tablet_service.cc:1467] Processing CreateTablet for tablet 87e3ffd99bd5437b831e454a03e5d68c (DEFAULT_TABLE table=test-table [id=4d04837408f848938c9a56a505295d7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:49.102053 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:49.088566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.13.148.131:38093)
0806 04:51:49.093006 (+ 4440us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:49.093015 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:49.093693 (+ 678us) server_negotiation.cc:407] Connection header received
0806 04:51:49.093799 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:49.093806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:49.093981 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:49.094131 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:49.094938 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.096338 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.098709 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.100109 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.100282 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:49.100418 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:49.101537 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0806 04:51:49.101679 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4286,"thread_start_us":4121,"threads_started":1}
I20260806 04:51:49.102684 18392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87e3ffd99bd5437b831e454a03e5d68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:49.104562 18256 tablet_service.cc:1467] Processing CreateTablet for tablet 87e3ffd99bd5437b831e454a03e5d68c (DEFAULT_TABLE table=test-table [id=4d04837408f848938c9a56a505295d7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:49.105185 18529 tablet_service.cc:1467] Processing CreateTablet for tablet 87e3ffd99bd5437b831e454a03e5d68c (DEFAULT_TABLE table=test-table [id=4d04837408f848938c9a56a505295d7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:49.105867 18256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87e3ffd99bd5437b831e454a03e5d68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:49.106247 18529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87e3ffd99bd5437b831e454a03e5d68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:49.117269 18613 tablet_bootstrap.cc:492] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8: Bootstrap starting.
I20260806 04:51:49.118063 18615 tablet_bootstrap.cc:492] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6: Bootstrap starting.
I20260806 04:51:49.119619 18616 tablet_bootstrap.cc:492] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6: Bootstrap starting.
I20260806 04:51:49.121654 18613 tablet_bootstrap.cc:654] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:49.122828 18616 tablet_bootstrap.cc:654] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:49.122808 18613 log.cc:824] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:49.123912 18616 log.cc:824] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:49.124181 18615 tablet_bootstrap.cc:654] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:49.125456 18615 log.cc:824] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:49.132511 18615 tablet_bootstrap.cc:492] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6: No bootstrap required, opened a new log
I20260806 04:51:49.132835 18616 tablet_bootstrap.cc:492] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6: No bootstrap required, opened a new log
I20260806 04:51:49.132872 18615 ts_tablet_manager.cc:1397] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260806 04:51:49.133050 18616 ts_tablet_manager.cc:1397] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260806 04:51:49.133095 18613 tablet_bootstrap.cc:492] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8: No bootstrap required, opened a new log
I20260806 04:51:49.133319 18613 ts_tablet_manager.cc:1397] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260806 04:51:49.140175 18616 raft_consensus.cc:348] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4147287d60c94b3b8ff64fa61f898bf6" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 34191 } } peers { permanent_uuid: "ca996479cbfa4a0fa504964c5fe06fc6" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 32829 } } peers { permanent_uuid: "4751b0386c4c4a96b2b3652cd1d8dca8" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 38093 } }
I20260806 04:51:49.140515 18616 raft_consensus.cc:374] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:49.140676 18616 raft_consensus.cc:729] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4147287d60c94b3b8ff64fa61f898bf6, State: Initialized, Role: FOLLOWER
I20260806 04:51:49.141880 18616 consensus_queue.cc:260] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4147287d60c94b3b8ff64fa61f898bf6" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 34191 } } peers { permanent_uuid: "ca996479cbfa4a0fa504964c5fe06fc6" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 32829 } } peers { permanent_uuid: "4751b0386c4c4a96b2b3652cd1d8dca8" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 38093 } }
I20260806 04:51:49.142935 18322 heartbeater.cc:503] Master 127.13.148.190:36751 was elected leader, sending a full tablet report...
I20260806 04:51:49.143148 18615 raft_consensus.cc:348] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4147287d60c94b3b8ff64fa61f898bf6" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 34191 } } peers { permanent_uuid: "ca996479cbfa4a0fa504964c5fe06fc6" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 32829 } } peers { permanent_uuid: "4751b0386c4c4a96b2b3652cd1d8dca8" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 38093 } }
I20260806 04:51:49.143290 18613 raft_consensus.cc:348] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4147287d60c94b3b8ff64fa61f898bf6" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 34191 } } peers { permanent_uuid: "ca996479cbfa4a0fa504964c5fe06fc6" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 32829 } } peers { permanent_uuid: "4751b0386c4c4a96b2b3652cd1d8dca8" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 38093 } }
I20260806 04:51:49.143558 18615 raft_consensus.cc:374] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:49.143613 18616 ts_tablet_manager.cc:1428] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260806 04:51:49.143652 18615 raft_consensus.cc:729] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca996479cbfa4a0fa504964c5fe06fc6, State: Initialized, Role: FOLLOWER
I20260806 04:51:49.143671 18613 raft_consensus.cc:374] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:49.143745 18613 raft_consensus.cc:729] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4751b0386c4c4a96b2b3652cd1d8dca8, State: Initialized, Role: FOLLOWER
I20260806 04:51:49.144224 18615 consensus_queue.cc:260] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4147287d60c94b3b8ff64fa61f898bf6" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 34191 } } peers { permanent_uuid: "ca996479cbfa4a0fa504964c5fe06fc6" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 32829 } } peers { permanent_uuid: "4751b0386c4c4a96b2b3652cd1d8dca8" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 38093 } }
I20260806 04:51:49.144224 18613 consensus_queue.cc:260] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4147287d60c94b3b8ff64fa61f898bf6" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 34191 } } peers { permanent_uuid: "ca996479cbfa4a0fa504964c5fe06fc6" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 32829 } } peers { permanent_uuid: "4751b0386c4c4a96b2b3652cd1d8dca8" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 38093 } }
I20260806 04:51:49.145370 18595 heartbeater.cc:503] Master 127.13.148.190:36751 was elected leader, sending a full tablet report...
I20260806 04:51:49.145637 18458 heartbeater.cc:503] Master 127.13.148.190:36751 was elected leader, sending a full tablet report...
I20260806 04:51:49.145809 18613 ts_tablet_manager.cc:1428] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20260806 04:51:49.146461 18615 ts_tablet_manager.cc:1428] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260806 04:51:49.192642 18620 raft_consensus.cc:482] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:51:49.192932 18620 raft_consensus.cc:504] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4147287d60c94b3b8ff64fa61f898bf6" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 34191 } } peers { permanent_uuid: "ca996479cbfa4a0fa504964c5fe06fc6" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 32829 } } peers { permanent_uuid: "4751b0386c4c4a96b2b3652cd1d8dca8" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 38093 } }
I20260806 04:51:49.194382 18620 leader_election.cc:288] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca996479cbfa4a0fa504964c5fe06fc6 (127.13.148.130:32829), 4751b0386c4c4a96b2b3652cd1d8dca8 (127.13.148.131:38093)
I20260806 04:51:49.200451 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:49.194764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:40347 (local address 127.13.148.130:32829)
0806 04:51:49.194986 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:49.194993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:49.195270 (+ 277us) server_negotiation.cc:407] Connection header received
0806 04:51:49.195475 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:49.195483 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:49.195570 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:49.195661 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:49.196315 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.197120 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.198782 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.199551 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.199720 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:49.199796 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:49.200070 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 04:51:49.200206 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:51:49.200539 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:49.194563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:32829 (local address 127.13.148.129:40347)
0806 04:51:49.195149 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:49.195164 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:49.195345 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:49.195798 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:49.195801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:49.195823 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:49.196163 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:49.196170 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.197240 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.197248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:49.198636 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:49.198647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.199758 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.199764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:49.199915 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:49.199938 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:51:49.200043 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 04:51:49.200198 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":312,"threads_started":1}
I20260806 04:51:49.200966 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:49.195324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:38093 (local address 127.13.148.129:40423)
0806 04:51:49.195901 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:49.195916 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:49.196065 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:49.196373 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:49.196377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:49.196398 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:51:49.196742 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:49.196750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.197805 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.197810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:49.199100 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:49.199110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.200261 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.200266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:49.200420 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:49.200439 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:51:49.200537 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 04:51:49.200651 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":265,"threads_started":1}
I20260806 04:51:49.201143 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:49.195512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:40423 (local address 127.13.148.131:38093)
0806 04:51:49.195704 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:49.195709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:49.195999 (+ 290us) server_negotiation.cc:407] Connection header received
0806 04:51:49.196192 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:49.196197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:49.196258 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:49.196356 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:51:49.196928 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.197646 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.199254 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.200021 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.200214 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:49.200305 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:49.200826 (+ 521us) server_negotiation.cc:299] Negotiation successful
0806 04:51:49.200935 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 04:51:49.201499 18412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e3ffd99bd5437b831e454a03e5d68c" candidate_uuid: "4147287d60c94b3b8ff64fa61f898bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca996479cbfa4a0fa504964c5fe06fc6" is_pre_election: true
I20260806 04:51:49.201985 18412 raft_consensus.cc:2454] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4147287d60c94b3b8ff64fa61f898bf6 in term 0.
I20260806 04:51:49.201984 18549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e3ffd99bd5437b831e454a03e5d68c" candidate_uuid: "4147287d60c94b3b8ff64fa61f898bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4751b0386c4c4a96b2b3652cd1d8dca8" is_pre_election: true
I20260806 04:51:49.202484 18549 raft_consensus.cc:2454] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4147287d60c94b3b8ff64fa61f898bf6 in term 0.
I20260806 04:51:49.202566 18208 leader_election.cc:302] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4147287d60c94b3b8ff64fa61f898bf6, ca996479cbfa4a0fa504964c5fe06fc6; no voters:
I20260806 04:51:49.202912 18620 raft_consensus.cc:2781] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:51:49.203047 18620 raft_consensus.cc:482] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:51:49.203150 18620 raft_consensus.cc:3037] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:49.205070 18620 raft_consensus.cc:504] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4147287d60c94b3b8ff64fa61f898bf6" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 34191 } } peers { permanent_uuid: "ca996479cbfa4a0fa504964c5fe06fc6" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 32829 } } peers { permanent_uuid: "4751b0386c4c4a96b2b3652cd1d8dca8" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 38093 } }
I20260806 04:51:49.205621 18620 leader_election.cc:288] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [CANDIDATE]: Term 1 election: Requested vote from peers ca996479cbfa4a0fa504964c5fe06fc6 (127.13.148.130:32829), 4751b0386c4c4a96b2b3652cd1d8dca8 (127.13.148.131:38093)
I20260806 04:51:49.206054 18412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e3ffd99bd5437b831e454a03e5d68c" candidate_uuid: "4147287d60c94b3b8ff64fa61f898bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca996479cbfa4a0fa504964c5fe06fc6"
I20260806 04:51:49.206092 18549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e3ffd99bd5437b831e454a03e5d68c" candidate_uuid: "4147287d60c94b3b8ff64fa61f898bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4751b0386c4c4a96b2b3652cd1d8dca8"
I20260806 04:51:49.206225 18412 raft_consensus.cc:3037] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:49.206255 18549 raft_consensus.cc:3037] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:49.208361 18412 raft_consensus.cc:2454] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4147287d60c94b3b8ff64fa61f898bf6 in term 1.
I20260806 04:51:49.208393 18549 raft_consensus.cc:2454] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4147287d60c94b3b8ff64fa61f898bf6 in term 1.
I20260806 04:51:49.208690 18208 leader_election.cc:302] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4147287d60c94b3b8ff64fa61f898bf6, 4751b0386c4c4a96b2b3652cd1d8dca8; no voters:
I20260806 04:51:49.208945 18620 raft_consensus.cc:2781] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:49.209350 18620 raft_consensus.cc:686] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [term 1 LEADER]: Becoming Leader. State: Replica: 4147287d60c94b3b8ff64fa61f898bf6, State: Running, Role: LEADER
I20260806 04:51:49.210064 18620 consensus_queue.cc:237] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4147287d60c94b3b8ff64fa61f898bf6" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 34191 } } peers { permanent_uuid: "ca996479cbfa4a0fa504964c5fe06fc6" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 32829 } } peers { permanent_uuid: "4751b0386c4c4a96b2b3652cd1d8dca8" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 38093 } }
I20260806 04:51:49.213845 18127 catalog_manager.cc:5495] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4147287d60c94b3b8ff64fa61f898bf6 (127.13.148.129). New cstate: current_term: 1 leader_uuid: "4147287d60c94b3b8ff64fa61f898bf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4147287d60c94b3b8ff64fa61f898bf6" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 34191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca996479cbfa4a0fa504964c5fe06fc6" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 32829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4751b0386c4c4a96b2b3652cd1d8dca8" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 38093 } health_report { overall_health: UNKNOWN } } }
I20260806 04:51:49.247502 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:49.242772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.13.148.129:34191)
0806 04:51:49.242948 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:49.242954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:49.243048 (+ 94us) server_negotiation.cc:407] Connection header received
0806 04:51:49.243260 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:49.243263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:49.243327 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:49.243466 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:51:49.243791 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.244388 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.245614 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.245978 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.246092 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:49.246521 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:51:49.246981 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:51:49.247137 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:49.247180 (+ 43us) server_negotiation.cc:299] Negotiation successful
0806 04:51:49.247267 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:51:49.254673 18549 raft_consensus.cc:1261] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8 [term 1 FOLLOWER]: Refusing update from remote peer 4147287d60c94b3b8ff64fa61f898bf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:49.254717 18412 raft_consensus.cc:1261] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6 [term 1 FOLLOWER]: Refusing update from remote peer 4147287d60c94b3b8ff64fa61f898bf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:49.255585 18625 consensus_queue.cc:1035] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4751b0386c4c4a96b2b3652cd1d8dca8" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 38093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:49.255964 18620 consensus_queue.cc:1035] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca996479cbfa4a0fa504964c5fe06fc6" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 32829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:49.265996 18628 mvcc.cc:205] Tried to move back new op lower bound from 7315422860299182080 to 7315422860127944704. Current Snapshot: MvccSnapshot[applied={T|T < 7315422860299182080}]
I20260806 04:51:49.279771 18128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:51:49.280068 18128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:51:49.283329 18128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08f0251a429343c397e5a8025984209a: Sending DeleteTablet for 3 replicas of tablet 87e3ffd99bd5437b831e454a03e5d68c
I20260806 04:51:49.284279 18256 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e3ffd99bd5437b831e454a03e5d68c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:49 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:54660
I20260806 04:51:49.284354 18392 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e3ffd99bd5437b831e454a03e5d68c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:49 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:35150
I20260806 04:51:49.285120 18529 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e3ffd99bd5437b831e454a03e5d68c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:49 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:54530
I20260806 04:51:49.285550 18640 tablet_replica.cc:330] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6: stopping tablet replica
I20260806 04:51:49.285931 18641 tablet_replica.cc:330] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8: stopping tablet replica
I20260806 04:51:49.285912 18640 raft_consensus.cc:2229] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:51:49.286020 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 18192
I20260806 04:51:49.286324 18640 raft_consensus.cc:2258] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:49.286296 18641 raft_consensus.cc:2229] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:51:49.287000 18639 tablet_replica.cc:330] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6: stopping tablet replica
I20260806 04:51:49.287518 18639 raft_consensus.cc:2229] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:51:49.288137 18641 raft_consensus.cc:2258] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:49.288146 18639 raft_consensus.cc:2258] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:49.289937 18640 ts_tablet_manager.cc:1905] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:49.291213 18641 ts_tablet_manager.cc:1905] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:49.295822 18639 ts_tablet_manager.cc:1905] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:49.297643 18641 ts_tablet_manager.cc:1918] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:51:49.297799 18641 log.cc:1196] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/87e3ffd99bd5437b831e454a03e5d68c
Aug 06 04:51:49 dist-test-slave-1x4g krb5kdc[18084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:51:49.298324 18641 ts_tablet_manager.cc:1939] T 87e3ffd99bd5437b831e454a03e5d68c P 4751b0386c4c4a96b2b3652cd1d8dca8: Deleting consensus metadata
I20260806 04:51:49.300365 18113 catalog_manager.cc:4841] TS 4751b0386c4c4a96b2b3652cd1d8dca8 (127.13.148.131:38093): tablet 87e3ffd99bd5437b831e454a03e5d68c (table test-table [id=4d04837408f848938c9a56a505295d7b]) successfully deleted
I20260806 04:51:49.302000 18640 ts_tablet_manager.cc:1918] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:51:49.302140 18640 log.cc:1196] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/87e3ffd99bd5437b831e454a03e5d68c
I20260806 04:51:49.302589 18640 ts_tablet_manager.cc:1939] T 87e3ffd99bd5437b831e454a03e5d68c P ca996479cbfa4a0fa504964c5fe06fc6: Deleting consensus metadata
I20260806 04:51:49.303476 18639 ts_tablet_manager.cc:1918] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:51:49.303618 18639 log.cc:1196] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/87e3ffd99bd5437b831e454a03e5d68c
I20260806 04:51:49.304019 18113 catalog_manager.cc:4841] TS ca996479cbfa4a0fa504964c5fe06fc6 (127.13.148.130:32829): tablet 87e3ffd99bd5437b831e454a03e5d68c (table test-table [id=4d04837408f848938c9a56a505295d7b]) successfully deleted
I20260806 04:51:49.304106 18639 ts_tablet_manager.cc:1939] T 87e3ffd99bd5437b831e454a03e5d68c P 4147287d60c94b3b8ff64fa61f898bf6: Deleting consensus metadata
I20260806 04:51:49.305480 18112 catalog_manager.cc:4841] TS 4147287d60c94b3b8ff64fa61f898bf6 (127.13.148.129:34191): tablet 87e3ffd99bd5437b831e454a03e5d68c (table test-table [id=4d04837408f848938c9a56a505295d7b]) successfully deleted
I20260806 04:51:49.306588 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:49.287250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54690 (local address 127.13.148.129:34191)
0806 04:51:49.287621 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:49.287629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:49.287948 (+ 319us) server_negotiation.cc:407] Connection header received
0806 04:51:49.288199 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:49.288214 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:49.288312 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:49.288386 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:49.290212 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.291229 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.293066 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.293660 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.293752 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:49.299357 (+ 5605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:49.299368 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:49.299382 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:49.299425 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:49.303416 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:49.304580 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:49.304588 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:49.304591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:49.304769 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:49.305219 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:49.305224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:49.305227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:49.305598 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:49.305904 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:49.306058 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 04:51:49.306243 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260806 04:51:49.307235 18317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:49.395316 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 18192
I20260806 04:51:49.413427 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 18328
Aug 06 04:51:49 dist-test-slave-1x4g krb5kdc[18084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:51:49.427484 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:49.414148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34310 (local address 127.13.148.130:32829)
0806 04:51:49.414317 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:49.414322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:49.414339 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:51:49.414467 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:49.414471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:49.414529 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:49.414614 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:49.415713 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.416455 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.417677 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.418064 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.418170 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:49.422436 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:49.422448 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:49.422462 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:49.422492 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:49.425130 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:49.425782 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:49.425789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:49.425791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:49.425916 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:49.426245 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:49.426248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:49.426249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:49.426576 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:49.426775 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:49.427004 (+ 229us) server_negotiation.cc:299] Negotiation successful
0806 04:51:49.427148 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 04:51:49.428179 18453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:49.510169 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 18328
I20260806 04:51:49.528599 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 18464
Aug 06 04:51:49 dist-test-slave-1x4g krb5kdc[18084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
I20260806 04:51:49.542847 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:49.529418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47746 (local address 127.13.148.131:38093)
0806 04:51:49.529600 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:49.529604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:49.529624 (+ 20us) server_negotiation.cc:407] Connection header received
0806 04:51:49.529747 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:49.529751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:49.529825 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:49.529927 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:49.531119 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.531768 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.533029 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:49.533396 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:49.533489 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:49.537465 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:49.537479 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:49.537494 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:49.537528 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:49.540142 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:49.540697 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:49.540704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:49.540706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:49.540922 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:49.541333 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:49.541337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:49.541340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:49.541659 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:49.542000 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:49.542330 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 04:51:49.542551 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 04:51:49.543536 18590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:49.625698 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 18464
I20260806 04:51:49.643682 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 18097
I20260806 04:51:49.644884 18160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:49.743835 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 18097
2026-08-06T04:51:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3745 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T04:51:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:51:49Z Disabled control of system clock
I20260806 04:51:49.810300 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:44111
--webserver_interface=127.13.148.190
--webserver_port=0
--builtin_ntp_servers=127.13.148.148:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:44111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 04:51:50.111848 18650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:50.112136 18650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:50.112241 18650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:50.118887 18650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:51:50.118995 18650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:50.119035 18650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:50.119103 18650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:51:50.119182 18650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:51:50.127188 18650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:44111
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:44111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:50.128322 18650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:50.130066 18650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:50.141362 18655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:50.141304 18656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:50.141367 18658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:50.142252 18650 server_base.cc:1034] running on GCE node
I20260806 04:51:50.142976 18650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:50.144361 18650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:50.145560 18650 hybrid_clock.cc:648] HybridClock initialized: now 1785991910145522 us; error 30 us; skew 500 ppm
I20260806 04:51:50.145877 18650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:51:50.147792 18650 webserver.cc:460] Webserver started at http://127.13.148.190:33793/ using document root <none> and password file <none>
I20260806 04:51:50.148427 18650 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:50.148536 18650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:50.148931 18650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:50.151654 18650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "2c19f5d0712f4f86afe8225354594c48"
format_stamp: "Formatted at 2026-08-06 04:51:50 on dist-test-slave-1x4g"
server_key: "4a6e8cb9741a05a4ea1a3ccf52a596af"
server_key_iv: "8a211cb1726836d4c9450be217a089b5"
server_key_version: "encryptionkey@0"
I20260806 04:51:50.152381 18650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "2c19f5d0712f4f86afe8225354594c48"
format_stamp: "Formatted at 2026-08-06 04:51:50 on dist-test-slave-1x4g"
server_key: "4a6e8cb9741a05a4ea1a3ccf52a596af"
server_key_iv: "8a211cb1726836d4c9450be217a089b5"
server_key_version: "encryptionkey@0"
I20260806 04:51:50.156986 18650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 04:51:50.160022 18664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:50.161269 18650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:51:50.161437 18650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "2c19f5d0712f4f86afe8225354594c48"
format_stamp: "Formatted at 2026-08-06 04:51:50 on dist-test-slave-1x4g"
server_key: "4a6e8cb9741a05a4ea1a3ccf52a596af"
server_key_iv: "8a211cb1726836d4c9450be217a089b5"
server_key_version: "encryptionkey@0"
I20260806 04:51:50.161633 18650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:50.166725 18650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:50.167343 18650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:50.167595 18650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:50.184185 18650 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:44111
I20260806 04:51:50.184222 18716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:44111 every 8 connection(s)
I20260806 04:51:50.185324 18650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:51:50.188376 18717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:50.194937 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 18650
I20260806 04:51:50.195132 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:51:50.195734 13906 external_mini_cluster.cc:1428] Setting key 6044a6935e302f8ec03016e5788fbc85
I20260806 04:51:50.197629 18717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48: Bootstrap starting.
I20260806 04:51:50.201462 18717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:50.202698 18717 log.cc:824] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:50.204485 18720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:50.197614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51020 (local address 127.13.148.190:44111)
0806 04:51:50.198604 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:50.198616 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:50.198661 (+ 45us) server_negotiation.cc:407] Connection header received
0806 04:51:50.200954 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:50.200978 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:50.201435 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:50.201874 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:50.202795 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:50.202802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:50.202816 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:50.202832 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:50.202896 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:50.202921 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:50.203090 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:50.203238 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 04:51:50.203509 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":261,"threads_started":1}
I20260806 04:51:50.205865 18717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48: No bootstrap required, opened a new log
I20260806 04:51:50.212378 18717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c19f5d0712f4f86afe8225354594c48" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44111 } }
I20260806 04:51:50.212738 18717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:50.212906 18717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c19f5d0712f4f86afe8225354594c48, State: Initialized, Role: FOLLOWER
I20260806 04:51:50.213414 18717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c19f5d0712f4f86afe8225354594c48" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44111 } }
I20260806 04:51:50.213763 18717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:51:50.213871 18717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:51:50.213989 18717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:50.215906 18717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c19f5d0712f4f86afe8225354594c48" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44111 } }
I20260806 04:51:50.216331 18717 leader_election.cc:302] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c19f5d0712f4f86afe8225354594c48; no voters:
I20260806 04:51:50.216926 18717 leader_election.cc:288] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:51:50.217029 18722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:50.217793 18722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [term 1 LEADER]: Becoming Leader. State: Replica: 2c19f5d0712f4f86afe8225354594c48, State: Running, Role: LEADER
I20260806 04:51:50.218297 18722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c19f5d0712f4f86afe8225354594c48" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44111 } }
I20260806 04:51:50.218967 18717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:51:50.221143 18724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c19f5d0712f4f86afe8225354594c48. Latest consensus state: current_term: 1 leader_uuid: "2c19f5d0712f4f86afe8225354594c48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c19f5d0712f4f86afe8225354594c48" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44111 } } }
I20260806 04:51:50.221117 18723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c19f5d0712f4f86afe8225354594c48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c19f5d0712f4f86afe8225354594c48" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44111 } } }
I20260806 04:51:50.221515 18724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:50.221550 18723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:50.223567 18729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:51:50.228410 18729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:51:50.236655 18729 catalog_manager.cc:1347] Generated new cluster ID: 6bac3f0f2d1d42718f502be4445a04c2
I20260806 04:51:50.236752 18729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:51:50.253829 18729 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:51:50.254220 18729 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:51:50.265199 18729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48: Generated new TSK 0
I20260806 04:51:50.265924 18729 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:51:50.274811 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44111
--builtin_ntp_servers=127.13.148.148:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 04:51:50.576267 18741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:50.576483 18741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:50.576575 18741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:50.583046 18741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:50.583128 18741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:50.583280 18741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:51:50.591629 18741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:50.592657 18741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:50.594310 18741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:50.606524 18750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:50.606637 18747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:50.606675 18746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:50.607493 18741 server_base.cc:1034] running on GCE node
I20260806 04:51:50.608013 18741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:50.608875 18741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:50.610092 18741 hybrid_clock.cc:648] HybridClock initialized: now 1785991910610050 us; error 36 us; skew 500 ppm
I20260806 04:51:50.610405 18741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:51:50.612342 18741 webserver.cc:460] Webserver started at http://127.13.148.129:43465/ using document root <none> and password file <none>
I20260806 04:51:50.613026 18741 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:50.613145 18741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:50.613430 18741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:50.615919 18741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "ae51203cbdc0440ab10a596deaf1055f"
format_stamp: "Formatted at 2026-08-06 04:51:50 on dist-test-slave-1x4g"
server_key: "14f593007b1aae6558d4653a8752ba69"
server_key_iv: "cf70483aa76cf04cab315de66b9d188b"
server_key_version: "encryptionkey@0"
I20260806 04:51:50.616693 18741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "ae51203cbdc0440ab10a596deaf1055f"
format_stamp: "Formatted at 2026-08-06 04:51:50 on dist-test-slave-1x4g"
server_key: "14f593007b1aae6558d4653a8752ba69"
server_key_iv: "cf70483aa76cf04cab315de66b9d188b"
server_key_version: "encryptionkey@0"
I20260806 04:51:50.621868 18741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 04:51:50.625082 18756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:50.626338 18741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:51:50.626492 18741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "ae51203cbdc0440ab10a596deaf1055f"
format_stamp: "Formatted at 2026-08-06 04:51:50 on dist-test-slave-1x4g"
server_key: "14f593007b1aae6558d4653a8752ba69"
server_key_iv: "cf70483aa76cf04cab315de66b9d188b"
server_key_version: "encryptionkey@0"
I20260806 04:51:50.626690 18741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:50.631969 18741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:50.632639 18741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:50.632937 18741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:50.633723 18741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:50.635054 18741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:50.635165 18741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:50.635294 18741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:50.635376 18741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:50.665485 18741 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:35315
I20260806 04:51:50.665551 18869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:35315 every 8 connection(s)
I20260806 04:51:50.666715 18741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:51:50.674795 18720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:50.669919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:59913 (local address 127.13.148.190:44111)
0806 04:51:50.670090 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:50.670094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:50.671340 (+ 1246us) server_negotiation.cc:407] Connection header received
0806 04:51:50.672787 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:50.672791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:50.672867 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:50.672956 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:50.674196 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:50.674200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:50.674203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:50.674214 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:50.674239 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:50.674249 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:50.674334 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:50.674481 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 04:51:50.674524 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 04:51:50.675099 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 18741
I20260806 04:51:50.675288 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:51:50.675693 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:50.670454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:44111 (local address 127.13.148.129:59913)
0806 04:51:50.671233 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:50.671263 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:50.672561 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:50.673143 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:50.673150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:50.673231 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:50.673956 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:51:50.673958 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:50.673993 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:51:50.674067 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:50.674325 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:50.674331 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:50.674336 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 04:51:50.674524 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 04:51:50.674805 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":240,"threads_started":1}
I20260806 04:51:50.675872 13906 external_mini_cluster.cc:1428] Setting key 3edfb92a5130844f72fe4f10ad789043
I20260806 04:51:50.677520 18870 heartbeater.cc:347] Connected to a master server at 127.13.148.190:44111
I20260806 04:51:50.677840 18870 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:50.678378 18870 heartbeater.cc:511] Master 127.13.148.190:44111 requested a full tablet report, sending...
I20260806 04:51:50.678910 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44111
--builtin_ntp_servers=127.13.148.148:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 04:51:50.680266 18681 ts_manager.cc:194] Registered new tserver with Master: ae51203cbdc0440ab10a596deaf1055f (127.13.148.129:35315)
W20260806 04:51:50.984252 18873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:50.984530 18873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:50.984673 18873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:50.991463 18873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:50.991595 18873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:50.991789 18873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:51:51.000416 18873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:51.001883 18873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:51.003612 18873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:51.017036 18878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:51.017086 18879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:51.016988 18881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:51.018141 18873 server_base.cc:1034] running on GCE node
I20260806 04:51:51.018626 18873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:51.019510 18873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:51.020746 18873 hybrid_clock.cc:648] HybridClock initialized: now 1785991911020706 us; error 52 us; skew 500 ppm
I20260806 04:51:51.021066 18873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:51:51.026203 18873 webserver.cc:460] Webserver started at http://127.13.148.130:46081/ using document root <none> and password file <none>
I20260806 04:51:51.026832 18873 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:51.026965 18873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:51.027256 18873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:51.030141 18873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "15ac670467a64be18e422849e9df1f46"
format_stamp: "Formatted at 2026-08-06 04:51:51 on dist-test-slave-1x4g"
server_key: "c849f2741f631419d1af3ba01b762556"
server_key_iv: "906566819c8796d8bc6fdb2a40ea28ce"
server_key_version: "encryptionkey@0"
I20260806 04:51:51.030963 18873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "15ac670467a64be18e422849e9df1f46"
format_stamp: "Formatted at 2026-08-06 04:51:51 on dist-test-slave-1x4g"
server_key: "c849f2741f631419d1af3ba01b762556"
server_key_iv: "906566819c8796d8bc6fdb2a40ea28ce"
server_key_version: "encryptionkey@0"
I20260806 04:51:51.035527 18873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 04:51:51.038800 18887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:51.040007 18873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:51:51.040179 18873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "15ac670467a64be18e422849e9df1f46"
format_stamp: "Formatted at 2026-08-06 04:51:51 on dist-test-slave-1x4g"
server_key: "c849f2741f631419d1af3ba01b762556"
server_key_iv: "906566819c8796d8bc6fdb2a40ea28ce"
server_key_version: "encryptionkey@0"
I20260806 04:51:51.040406 18873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:51.045450 18873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:51.046212 18873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:51.046473 18873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:51.047286 18873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:51.048482 18873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:51.048615 18873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:51.048739 18873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:51.048832 18873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 04:51:51.079269 18873 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:34889
I20260806 04:51:51.079350 19001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:34889 every 8 connection(s)
I20260806 04:51:51.080566 18873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:51:51.088464 18720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.082946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:35553 (local address 127.13.148.190:44111)
0806 04:51:51.083145 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:51.083161 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:51.084350 (+ 1189us) server_negotiation.cc:407] Connection header received
0806 04:51:51.086363 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:51.086368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:51.086439 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:51.086546 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:51.087693 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:51.087699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:51.087703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:51.087723 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:51.087760 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:51.087771 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:51.087907 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:51.088097 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 04:51:51.088165 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 04:51:51.088472 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 18873
I20260806 04:51:51.088660 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:51:51.089092 13906 external_mini_cluster.cc:1428] Setting key e263d85e35493e33fb85118a315c0f7c
I20260806 04:51:51.089426 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.083404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:44111 (local address 127.13.148.130:35553)
0806 04:51:51.084212 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:51.084243 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:51.086095 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:51.086762 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:51.086769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:51.086851 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:51.087430 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:51:51.087432 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:51.087467 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:51:51.087541 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:51.087866 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:51.087877 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:51.087887 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 04:51:51.088173 (+ 286us) client_negotiation.cc:240] Negotiation successful
0806 04:51:51.088395 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":251,"threads_started":1}
I20260806 04:51:51.090827 19002 heartbeater.cc:347] Connected to a master server at 127.13.148.190:44111
I20260806 04:51:51.091114 19002 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:51.091642 19002 heartbeater.cc:511] Master 127.13.148.190:44111 requested a full tablet report, sending...
I20260806 04:51:51.091964 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44111
--builtin_ntp_servers=127.13.148.148:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 04:51:51.093029 18682 ts_manager.cc:194] Registered new tserver with Master: 15ac670467a64be18e422849e9df1f46 (127.13.148.130:34889)
W20260806 04:51:51.392448 19005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:51.392689 19005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:51.392771 19005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:51.399562 19005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:51.399649 19005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:51.399760 19005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:51:51.407944 19005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:51.409147 19005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:51.410852 19005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:51.423080 19011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:51.423110 19012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:51.423321 19014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:51.424108 19005 server_base.cc:1034] running on GCE node
I20260806 04:51:51.424640 19005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:51.425558 19005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:51.426769 19005 hybrid_clock.cc:648] HybridClock initialized: now 1785991911426737 us; error 43 us; skew 500 ppm
I20260806 04:51:51.427055 19005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:51:51.429328 19005 webserver.cc:460] Webserver started at http://127.13.148.131:33715/ using document root <none> and password file <none>
I20260806 04:51:51.429997 19005 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:51.430121 19005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:51.430523 19005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:51.433192 19005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "c6cd1e2316af420984c9ecf237869694"
format_stamp: "Formatted at 2026-08-06 04:51:51 on dist-test-slave-1x4g"
server_key: "e764feb8c09c5b145b78cbf5e8e8076e"
server_key_iv: "e4335005cbb9b164542766b2b6e44af0"
server_key_version: "encryptionkey@0"
I20260806 04:51:51.433985 19005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "c6cd1e2316af420984c9ecf237869694"
format_stamp: "Formatted at 2026-08-06 04:51:51 on dist-test-slave-1x4g"
server_key: "e764feb8c09c5b145b78cbf5e8e8076e"
server_key_iv: "e4335005cbb9b164542766b2b6e44af0"
server_key_version: "encryptionkey@0"
I20260806 04:51:51.438808 19005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:51:51.441900 19020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:51.443023 19005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:51:51.443171 19005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "c6cd1e2316af420984c9ecf237869694"
format_stamp: "Formatted at 2026-08-06 04:51:51 on dist-test-slave-1x4g"
server_key: "e764feb8c09c5b145b78cbf5e8e8076e"
server_key_iv: "e4335005cbb9b164542766b2b6e44af0"
server_key_version: "encryptionkey@0"
I20260806 04:51:51.443356 19005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:51.448458 19005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:51.449187 19005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:51.449467 19005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:51.450323 19005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:51.451638 19005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:51.451751 19005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:51.451859 19005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:51.451934 19005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:51.482844 19005 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:37727
I20260806 04:51:51.482890 19133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:37727 every 8 connection(s)
I20260806 04:51:51.484055 19005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:51:51.490478 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 19005
I20260806 04:51:51.490653 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:51:51.491032 13906 external_mini_cluster.cc:1428] Setting key cd4ed492eab6713e7152e1dfc2c22d44
I20260806 04:51:51.492045 18720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.486436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:37947 (local address 127.13.148.190:44111)
0806 04:51:51.486617 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:51.486621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:51.487935 (+ 1314us) server_negotiation.cc:407] Connection header received
0806 04:51:51.489775 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:51.489780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:51.489834 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:51.489920 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:51.491289 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:51.491295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:51.491301 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:51.491321 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:51.491364 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:51.491391 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:51.491522 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:51.491721 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 04:51:51.491787 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:51:51.493038 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.486882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:44111 (local address 127.13.148.131:37947)
0806 04:51:51.487769 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:51.487815 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:51.489530 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:51.490127 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:51.490134 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:51.490221 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:51.490938 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:51:51.490941 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:51.490981 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:51:51.491077 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:51.491503 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:51.491510 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:51.491515 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 04:51:51.491796 (+ 281us) client_negotiation.cc:240] Negotiation successful
0806 04:51:51.492015 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":233,"threads_started":1}
I20260806 04:51:51.494230 19134 heartbeater.cc:347] Connected to a master server at 127.13.148.190:44111
I20260806 04:51:51.494498 19134 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:51.495007 19134 heartbeater.cc:511] Master 127.13.148.190:44111 requested a full tablet report, sending...
I20260806 04:51:51.496181 18682 ts_manager.cc:194] Registered new tserver with Master: c6cd1e2316af420984c9ecf237869694 (127.13.148.131:37727)
I20260806 04:51:51.506111 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:51:51.512952 18720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.510418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.13.148.190:44111)
0806 04:51:51.510564 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:51.510568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:51.511067 (+ 499us) server_negotiation.cc:407] Connection header received
0806 04:51:51.511240 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:51.511251 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:51.511294 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:51.511386 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:51.512356 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:51.512361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:51.512365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:51.512381 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:51.512416 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:51.512427 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:51.512547 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:51.512639 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 04:51:51.512693 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:51:51.516409 18682 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:51:51.523499 18682 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:51:51.538877 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.533830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:34889 (local address 127.0.0.1:52988)
0806 04:51:51.534437 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:51.534497 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:51.534603 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:51.535696 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:51.535706 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:51.535801 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:51.536650 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:51:51.536653 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:51.536679 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:51:51.536782 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:51.537933 (+ 1151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:51.537948 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:51.537956 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 04:51:51.538103 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 04:51:51.538198 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":291,"threads_started":1}
I20260806 04:51:51.540491 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.533991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52988 (local address 127.13.148.130:34889)
0806 04:51:51.535083 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:51.535092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:51.535129 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:51:51.535207 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:51.535217 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:51.535466 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:51.536302 (+ 836us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:51.537633 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:51.537644 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:51.537661 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:51.537682 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:51.537743 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:51.537771 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:51.539818 (+ 2047us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:51.539898 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 04:51:51.540022 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962,"thread_start_us":2514,"threads_started":1}
I20260806 04:51:51.543128 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.533978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:37727 (local address 127.0.0.1:42468)
0806 04:51:51.539414 (+ 5436us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:51.539436 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:51.539554 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:51.540184 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:51.540192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:51.540251 (+ 59us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:51.540819 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:51:51.540824 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:51.540843 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:51:51.540898 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:51.542594 (+ 1696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:51.542602 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:51.542609 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 04:51:51.542694 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 04:51:51.542751 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5304,"threads_started":1}
I20260806 04:51:51.545083 18937 tablet_service.cc:1467] Processing CreateTablet for tablet 7b5935471b014590bcd3b0914e2400a6 (DEFAULT_TABLE table=test-table [id=28020bed438b46c1be29b1462ba1811f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:51.546831 18937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5935471b014590bcd3b0914e2400a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:51.547636 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.535785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33266 (local address 127.13.148.129:35315)
0806 04:51:51.536282 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:51.536290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:51.543763 (+ 7473us) server_negotiation.cc:407] Connection header received
0806 04:51:51.543847 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:51.543856 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:51.544091 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:51.544301 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:51.546222 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:51.546234 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:51.546249 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:51.546280 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:51.546342 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:51.546364 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:51.546482 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:51.547244 (+ 762us) server_negotiation.cc:299] Negotiation successful
0806 04:51:51.547347 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":257,"threads_started":1}
I20260806 04:51:51.548079 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.534488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42468 (local address 127.13.148.131:37727)
0806 04:51:51.535040 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:51.535047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:51.539566 (+ 4519us) server_negotiation.cc:407] Connection header received
0806 04:51:51.539690 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:51.539702 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:51.539958 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:51.540547 (+ 589us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:51.542335 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:51.542345 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:51.542360 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:51.542379 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:51.542439 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:51.542456 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:51.545691 (+ 3235us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:51.547606 (+ 1915us) server_negotiation.cc:299] Negotiation successful
0806 04:51:51.547695 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":1059,"threads_started":1}
I20260806 04:51:51.548698 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.535261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:35315 (local address 127.0.0.1:33266)
0806 04:51:51.543532 (+ 8271us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:51.543551 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:51.543640 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:51.545282 (+ 1642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:51.545289 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:51.545319 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:51.545897 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:51:51.545902 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:51.545985 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:51:51.546041 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:51.546533 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:51.546539 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:51.546546 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 04:51:51.546649 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 04:51:51.546699 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8181,"thread_start_us":11660,"threads_started":1}
I20260806 04:51:51.552428 18805 tablet_service.cc:1467] Processing CreateTablet for tablet 7b5935471b014590bcd3b0914e2400a6 (DEFAULT_TABLE table=test-table [id=28020bed438b46c1be29b1462ba1811f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:51.553757 18805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5935471b014590bcd3b0914e2400a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:51.553907 19069 tablet_service.cc:1467] Processing CreateTablet for tablet 7b5935471b014590bcd3b0914e2400a6 (DEFAULT_TABLE table=test-table [id=28020bed438b46c1be29b1462ba1811f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:51:51.555266 19069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5935471b014590bcd3b0914e2400a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:51.560989 19154 tablet_bootstrap.cc:492] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46: Bootstrap starting.
I20260806 04:51:51.564306 19154 tablet_bootstrap.cc:654] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:51.564963 19155 tablet_bootstrap.cc:492] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f: Bootstrap starting.
I20260806 04:51:51.565769 19154 log.cc:824] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:51.568259 19157 tablet_bootstrap.cc:492] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694: Bootstrap starting.
I20260806 04:51:51.568298 19154 tablet_bootstrap.cc:492] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46: No bootstrap required, opened a new log
I20260806 04:51:51.568374 19155 tablet_bootstrap.cc:654] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:51.569016 19154 ts_tablet_manager.cc:1397] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 04:51:51.569389 19155 log.cc:824] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:51.571535 19157 tablet_bootstrap.cc:654] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:51.572301 19155 tablet_bootstrap.cc:492] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f: No bootstrap required, opened a new log
I20260806 04:51:51.572530 19155 ts_tablet_manager.cc:1397] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 04:51:51.572561 19157 log.cc:824] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:51.576766 19154 raft_consensus.cc:348] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ac670467a64be18e422849e9df1f46" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 34889 } } peers { permanent_uuid: "c6cd1e2316af420984c9ecf237869694" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37727 } } peers { permanent_uuid: "ae51203cbdc0440ab10a596deaf1055f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35315 } }
I20260806 04:51:51.577191 19154 raft_consensus.cc:374] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:51.577332 19154 raft_consensus.cc:729] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ac670467a64be18e422849e9df1f46, State: Initialized, Role: FOLLOWER
I20260806 04:51:51.577968 19154 consensus_queue.cc:260] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ac670467a64be18e422849e9df1f46" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 34889 } } peers { permanent_uuid: "c6cd1e2316af420984c9ecf237869694" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37727 } } peers { permanent_uuid: "ae51203cbdc0440ab10a596deaf1055f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35315 } }
I20260806 04:51:51.578894 19157 tablet_bootstrap.cc:492] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694: No bootstrap required, opened a new log
I20260806 04:51:51.579190 19157 ts_tablet_manager.cc:1397] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260806 04:51:51.579664 19154 ts_tablet_manager.cc:1428] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260806 04:51:51.579819 19002 heartbeater.cc:503] Master 127.13.148.190:44111 was elected leader, sending a full tablet report...
I20260806 04:51:51.581666 19155 raft_consensus.cc:348] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ac670467a64be18e422849e9df1f46" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 34889 } } peers { permanent_uuid: "c6cd1e2316af420984c9ecf237869694" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37727 } } peers { permanent_uuid: "ae51203cbdc0440ab10a596deaf1055f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35315 } }
I20260806 04:51:51.582031 19155 raft_consensus.cc:374] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:51.582141 19155 raft_consensus.cc:729] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae51203cbdc0440ab10a596deaf1055f, State: Initialized, Role: FOLLOWER
W20260806 04:51:51.582242 19003 tablet.cc:2369] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:51:51.582830 19155 consensus_queue.cc:260] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ac670467a64be18e422849e9df1f46" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 34889 } } peers { permanent_uuid: "c6cd1e2316af420984c9ecf237869694" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37727 } } peers { permanent_uuid: "ae51203cbdc0440ab10a596deaf1055f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35315 } }
I20260806 04:51:51.583890 18870 heartbeater.cc:503] Master 127.13.148.190:44111 was elected leader, sending a full tablet report...
I20260806 04:51:51.584420 19155 ts_tablet_manager.cc:1428] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260806 04:51:51.587576 19157 raft_consensus.cc:348] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ac670467a64be18e422849e9df1f46" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 34889 } } peers { permanent_uuid: "c6cd1e2316af420984c9ecf237869694" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37727 } } peers { permanent_uuid: "ae51203cbdc0440ab10a596deaf1055f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35315 } }
I20260806 04:51:51.587973 19157 raft_consensus.cc:374] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:51.588080 19157 raft_consensus.cc:729] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6cd1e2316af420984c9ecf237869694, State: Initialized, Role: FOLLOWER
I20260806 04:51:51.588572 19157 consensus_queue.cc:260] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ac670467a64be18e422849e9df1f46" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 34889 } } peers { permanent_uuid: "c6cd1e2316af420984c9ecf237869694" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37727 } } peers { permanent_uuid: "ae51203cbdc0440ab10a596deaf1055f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35315 } }
I20260806 04:51:51.589596 19134 heartbeater.cc:503] Master 127.13.148.190:44111 was elected leader, sending a full tablet report...
I20260806 04:51:51.590159 19157 ts_tablet_manager.cc:1428] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260806 04:51:51.619525 19161 raft_consensus.cc:482] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:51:51.619745 19161 raft_consensus.cc:504] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ac670467a64be18e422849e9df1f46" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 34889 } } peers { permanent_uuid: "c6cd1e2316af420984c9ecf237869694" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37727 } } peers { permanent_uuid: "ae51203cbdc0440ab10a596deaf1055f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35315 } }
I20260806 04:51:51.621275 19161 leader_election.cc:288] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ac670467a64be18e422849e9df1f46 (127.13.148.130:34889), c6cd1e2316af420984c9ecf237869694 (127.13.148.131:37727)
I20260806 04:51:51.624493 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.621745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:40033 (local address 127.13.148.131:37727)
0806 04:51:51.621940 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:51.621961 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:51.622600 (+ 639us) server_negotiation.cc:407] Connection header received
0806 04:51:51.622780 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:51.622787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:51.622835 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:51.622925 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:51.623918 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:51.623923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:51.623927 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:51.623950 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:51.623986 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:51.623998 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:51.624114 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:51.624211 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 04:51:51.624269 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 04:51:51.624634 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.621612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:58717 (local address 127.13.148.130:34889)
0806 04:51:51.621836 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:51.621845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:51.622760 (+ 915us) server_negotiation.cc:407] Connection header received
0806 04:51:51.622914 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:51.622921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:51.623003 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:51.623104 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:51.623997 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:51.624002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:51.624006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:51.624027 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:51.624054 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:51.624083 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:51.624198 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:51.624348 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 04:51:51.624403 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 04:51:51.624728 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.621587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:37727 (local address 127.13.148.129:40033)
0806 04:51:51.622483 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:51.622505 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:51.622642 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:51.622946 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:51.622950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:51.622975 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:51.623701 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:51:51.623705 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:51.623723 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:51:51.623773 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:51.624102 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:51.624107 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:51.624111 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 04:51:51.624253 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 04:51:51.624296 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":751,"threads_started":1}
I20260806 04:51:51.624915 19163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.621609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:34889 (local address 127.13.148.129:58717)
0806 04:51:51.622656 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:51.622668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:51.622796 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:51.623137 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:51.623143 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:51.623162 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:51.623828 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:51:51.623831 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:51.623842 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:51:51.623874 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:51.624240 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:51.624243 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:51.624246 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 04:51:51.624380 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 04:51:51.624424 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"thread_start_us":437,"threads_started":1}
I20260806 04:51:51.625617 18957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5935471b014590bcd3b0914e2400a6" candidate_uuid: "ae51203cbdc0440ab10a596deaf1055f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ac670467a64be18e422849e9df1f46" is_pre_election: true
I20260806 04:51:51.625583 19089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5935471b014590bcd3b0914e2400a6" candidate_uuid: "ae51203cbdc0440ab10a596deaf1055f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6cd1e2316af420984c9ecf237869694" is_pre_election: true
I20260806 04:51:51.625955 18957 raft_consensus.cc:2454] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae51203cbdc0440ab10a596deaf1055f in term 0.
I20260806 04:51:51.625974 19089 raft_consensus.cc:2454] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae51203cbdc0440ab10a596deaf1055f in term 0.
I20260806 04:51:51.626443 18759 leader_election.cc:302] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ac670467a64be18e422849e9df1f46, ae51203cbdc0440ab10a596deaf1055f; no voters:
I20260806 04:51:51.626792 19161 raft_consensus.cc:2781] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:51:51.626924 19161 raft_consensus.cc:482] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:51:51.627015 19161 raft_consensus.cc:3037] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:51.629014 19161 raft_consensus.cc:504] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ac670467a64be18e422849e9df1f46" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 34889 } } peers { permanent_uuid: "c6cd1e2316af420984c9ecf237869694" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37727 } } peers { permanent_uuid: "ae51203cbdc0440ab10a596deaf1055f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35315 } }
I20260806 04:51:51.629561 19161 leader_election.cc:288] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [CANDIDATE]: Term 1 election: Requested vote from peers 15ac670467a64be18e422849e9df1f46 (127.13.148.130:34889), c6cd1e2316af420984c9ecf237869694 (127.13.148.131:37727)
I20260806 04:51:51.629891 18957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5935471b014590bcd3b0914e2400a6" candidate_uuid: "ae51203cbdc0440ab10a596deaf1055f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ac670467a64be18e422849e9df1f46"
I20260806 04:51:51.630072 18957 raft_consensus.cc:3037] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:51.630064 19089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5935471b014590bcd3b0914e2400a6" candidate_uuid: "ae51203cbdc0440ab10a596deaf1055f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6cd1e2316af420984c9ecf237869694"
I20260806 04:51:51.630244 19089 raft_consensus.cc:3037] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:51.632206 18957 raft_consensus.cc:2454] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae51203cbdc0440ab10a596deaf1055f in term 1.
I20260806 04:51:51.632382 19089 raft_consensus.cc:2454] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae51203cbdc0440ab10a596deaf1055f in term 1.
I20260806 04:51:51.632540 18759 leader_election.cc:302] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ac670467a64be18e422849e9df1f46, ae51203cbdc0440ab10a596deaf1055f; no voters:
I20260806 04:51:51.632874 19161 raft_consensus.cc:2781] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:51.633263 19161 raft_consensus.cc:686] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [term 1 LEADER]: Becoming Leader. State: Replica: ae51203cbdc0440ab10a596deaf1055f, State: Running, Role: LEADER
I20260806 04:51:51.633668 19161 consensus_queue.cc:237] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ac670467a64be18e422849e9df1f46" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 34889 } } peers { permanent_uuid: "c6cd1e2316af420984c9ecf237869694" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37727 } } peers { permanent_uuid: "ae51203cbdc0440ab10a596deaf1055f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35315 } }
I20260806 04:51:51.638044 18682 catalog_manager.cc:5495] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f reported cstate change: term changed from 0 to 1, leader changed from <none> to ae51203cbdc0440ab10a596deaf1055f (127.13.148.129). New cstate: current_term: 1 leader_uuid: "ae51203cbdc0440ab10a596deaf1055f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ac670467a64be18e422849e9df1f46" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 34889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6cd1e2316af420984c9ecf237869694" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae51203cbdc0440ab10a596deaf1055f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35315 } health_report { overall_health: HEALTHY } } }
I20260806 04:51:51.669188 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.667150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33270 (local address 127.13.148.129:35315)
0806 04:51:51.667329 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:51.667334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:51.667466 (+ 132us) server_negotiation.cc:407] Connection header received
0806 04:51:51.667613 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:51.667618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:51.667692 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:51.667781 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:51.668549 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:51.668555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:51.668559 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:51.668575 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:51.668607 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:51.668636 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:51.668734 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:51.668872 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 04:51:51.668921 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 04:51:51.669656 18871 tablet.cc:2369] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:51:51.677234 19089 raft_consensus.cc:1261] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694 [term 1 FOLLOWER]: Refusing update from remote peer ae51203cbdc0440ab10a596deaf1055f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:51.677281 18957 raft_consensus.cc:1261] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46 [term 1 FOLLOWER]: Refusing update from remote peer ae51203cbdc0440ab10a596deaf1055f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:51:51.678097 19165 consensus_queue.cc:1035] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6cd1e2316af420984c9ecf237869694" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:51.678422 19161 consensus_queue.cc:1035] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ac670467a64be18e422849e9df1f46" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 34889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:51:51.688318 19168 mvcc.cc:205] Tried to move back new op lower bound from 7315422870221209600 to 7315422870055108608. Current Snapshot: MvccSnapshot[applied={T|T < 7315422870221209600}]
I20260806 04:51:51.690690 19170 mvcc.cc:205] Tried to move back new op lower bound from 7315422870221209600 to 7315422870055108608. Current Snapshot: MvccSnapshot[applied={T|T < 7315422870221209600}]
I20260806 04:51:51.690953 19169 mvcc.cc:205] Tried to move back new op lower bound from 7315422870221209600 to 7315422870055108608. Current Snapshot: MvccSnapshot[applied={T|T < 7315422870221209600}]
I20260806 04:51:51.701217 18682 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:51:51.701429 18682 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:51:51.704553 18682 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c19f5d0712f4f86afe8225354594c48: Sending DeleteTablet for 3 replicas of tablet 7b5935471b014590bcd3b0914e2400a6
I20260806 04:51:51.705498 18937 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b5935471b014590bcd3b0914e2400a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:51 UTC) from {username='slave'} at 127.0.0.1:52988
I20260806 04:51:51.705574 18805 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b5935471b014590bcd3b0914e2400a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:51 UTC) from {username='slave'} at 127.0.0.1:33266
I20260806 04:51:51.705577 19069 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b5935471b014590bcd3b0914e2400a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:51:51 UTC) from {username='slave'} at 127.0.0.1:42468
I20260806 04:51:51.706499 19178 tablet_replica.cc:330] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f: stopping tablet replica
I20260806 04:51:51.706914 19178 raft_consensus.cc:2229] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:51:51.707242 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 18741
I20260806 04:51:51.707677 19178 raft_consensus.cc:2258] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:51.709872 19178 ts_tablet_manager.cc:1905] T 7b5935471b014590bcd3b0914e2400a6 P ae51203cbdc0440ab10a596deaf1055f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:51.710464 19177 tablet_replica.cc:330] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46: stopping tablet replica
I20260806 04:51:51.711042 19179 tablet_replica.cc:330] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694: stopping tablet replica
I20260806 04:51:51.711694 19177 raft_consensus.cc:2229] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:51:51.712097 19177 raft_consensus.cc:2258] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:51.713397 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.709675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.13.148.129:35315)
0806 04:51:51.709823 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:51.709829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:51.711235 (+ 1406us) server_negotiation.cc:407] Connection header received
0806 04:51:51.711411 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:51.711416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:51.711467 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:51.711578 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:51.712600 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:51.712607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:51.712615 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:51.712636 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:51.712682 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:51.712699 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:51.712808 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:51.713078 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 04:51:51.713130 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:51:51.712131 19179 raft_consensus.cc:2229] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:51:51.713881 19177 ts_tablet_manager.cc:1905] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:51.714040 19179 raft_consensus.cc:2258] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:51:51.714304 18865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:51.716518 19179 ts_tablet_manager.cc:1905] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:51:51.720857 19177 ts_tablet_manager.cc:1918] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:51:51.721000 19177 log.cc:1196] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/7b5935471b014590bcd3b0914e2400a6
I20260806 04:51:51.721478 19177 ts_tablet_manager.cc:1939] T 7b5935471b014590bcd3b0914e2400a6 P 15ac670467a64be18e422849e9df1f46: Deleting consensus metadata
I20260806 04:51:51.722349 19179 ts_tablet_manager.cc:1918] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:51:51.722503 19179 log.cc:1196] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/7b5935471b014590bcd3b0914e2400a6
I20260806 04:51:51.723063 19179 ts_tablet_manager.cc:1939] T 7b5935471b014590bcd3b0914e2400a6 P c6cd1e2316af420984c9ecf237869694: Deleting consensus metadata
I20260806 04:51:51.723024 18667 catalog_manager.cc:4841] TS 15ac670467a64be18e422849e9df1f46 (127.13.148.130:34889): tablet 7b5935471b014590bcd3b0914e2400a6 (table test-table [id=28020bed438b46c1be29b1462ba1811f]) successfully deleted
I20260806 04:51:51.724368 18665 catalog_manager.cc:4841] TS c6cd1e2316af420984c9ecf237869694 (127.13.148.131:37727): tablet 7b5935471b014590bcd3b0914e2400a6 (table test-table [id=28020bed438b46c1be29b1462ba1811f]) successfully deleted
I20260806 04:51:51.803848 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 18741
W20260806 04:51:51.821902 18666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.129:35315 (error 108)
I20260806 04:51:51.822182 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 18873
I20260806 04:51:51.824057 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.823373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:35315 (local address 127.0.0.1:33290)
0806 04:51:51.823626 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:51.823815 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:35315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260806 04:51:51.824329 18666 catalog_manager.cc:4584] TS ae51203cbdc0440ab10a596deaf1055f (127.13.148.129:35315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b5935471b014590bcd3b0914e2400a6: Network error: Client connection negotiation failed: client connection to 127.13.148.129:35315: connect: Connection refused (error 111)
I20260806 04:51:51.825150 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.822931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.13.148.130:34889)
0806 04:51:51.823110 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:51.823133 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:51.823154 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:51:51.823350 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:51.823356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:51.823502 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:51.823616 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:51.824568 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:51.824574 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:51.824578 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:51.824608 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:51.824645 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:51.824658 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:51.824756 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:51.824912 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 04:51:51.824958 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 04:51:51.825865 18997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:51.878037 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.877617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:35315 (local address 127.0.0.1:33302)
0806 04:51:51.877809 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:51.877914 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:35315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 04:51:51.878942 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.878610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:35315 (local address 127.0.0.1:33308)
0806 04:51:51.878755 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:51.878821 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:35315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 04:51:51.905059 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 18873
I20260806 04:51:51.924960 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 19005
I20260806 04:51:51.927286 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.925687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42474 (local address 127.13.148.131:37727)
0806 04:51:51.925835 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:51.925839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:51.925854 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:51:51.925907 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:51.925911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:51.926000 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:51.926075 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:51.926851 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:51.926854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:51.926857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:51.926867 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:51.926892 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:51.926900 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:51.926992 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:51.927045 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 04:51:51.927087 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:51:51.927896 19129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:51.939309 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.938762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:35315 (local address 127.0.0.1:33324)
0806 04:51:51.939025 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:51.939121 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:35315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260806 04:51:51.940284 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:51.939969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:35315 (local address 127.0.0.1:33340)
0806 04:51:51.940101 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:51.940164 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:35315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 04:51:52.006769 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 19005
I20260806 04:51:52.020991 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:52.020552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:35315 (local address 127.0.0.1:33342)
0806 04:51:52.020723 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:52.020850 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:35315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 04:51:52.022029 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:52.021672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:35315 (local address 127.0.0.1:33352)
0806 04:51:52.021840 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:52.021910 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:35315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 04:51:52.025554 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 18650
I20260806 04:51:52.026526 18712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:52.125070 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 18650
2026-08-06T04:51:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2379 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 04:51:52.166880 13906 mini_oidc.cc:152] Starting JWKS server
I20260806 04:51:52.169576 13906 webserver.cc:460] Webserver started at https://127.0.0.1:36071/ using document root <none> and password file <none>
I20260806 04:51:52.169883 13906 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 04:51:52.170529 13906 webserver.cc:460] Webserver started at http://127.0.0.1:36165/ using document root <none> and password file <none>
2026-08-06T04:51:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:51:52Z Disabled control of system clock
I20260806 04:51:52.197995 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:37479
--webserver_interface=127.13.148.190
--webserver_port=0
--builtin_ntp_servers=127.13.148.148:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:37479
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36071/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/testchainca.cert with env {}
W20260806 04:51:52.499410 19191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:52.499686 19191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:52.499778 19191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:52.506405 19191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:51:52.506501 19191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:52.506539 19191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:52.506595 19191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:51:52.506631 19191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 04:51:52.506691 19191 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 04:51:52.506731 19191 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36071/jwks.json
I20260806 04:51:52.515756 19191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:37479
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:37479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36071/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:52.516894 19191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:52.518071 19191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:52.529215 19197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:52.529170 19198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:52.529197 19200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:52.529902 19191 server_base.cc:1034] running on GCE node
I20260806 04:51:52.530643 19191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:52.531769 19191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:52.532999 19191 hybrid_clock.cc:648] HybridClock initialized: now 1785991912532949 us; error 27 us; skew 500 ppm
I20260806 04:51:52.533315 19191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:51:52.540004 19191 webserver.cc:460] Webserver started at http://127.13.148.190:39845/ using document root <none> and password file <none>
I20260806 04:51:52.540704 19191 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:52.540853 19191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:52.541152 19191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:52.543915 19191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "ce23ab0ba7164fcaa601e81cfb2db5d7"
format_stamp: "Formatted at 2026-08-06 04:51:52 on dist-test-slave-1x4g"
server_key: "7cebe9dae3e0d23675801b4eaa9ffc3c"
server_key_iv: "0d270576cd2dbff7fcb109e13719d2ba"
server_key_version: "encryptionkey@0"
I20260806 04:51:52.544682 19191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "ce23ab0ba7164fcaa601e81cfb2db5d7"
format_stamp: "Formatted at 2026-08-06 04:51:52 on dist-test-slave-1x4g"
server_key: "7cebe9dae3e0d23675801b4eaa9ffc3c"
server_key_iv: "0d270576cd2dbff7fcb109e13719d2ba"
server_key_version: "encryptionkey@0"
I20260806 04:51:52.549423 19191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 04:51:52.552600 19206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:52.553962 19191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:51:52.554155 19191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "ce23ab0ba7164fcaa601e81cfb2db5d7"
format_stamp: "Formatted at 2026-08-06 04:51:52 on dist-test-slave-1x4g"
server_key: "7cebe9dae3e0d23675801b4eaa9ffc3c"
server_key_iv: "0d270576cd2dbff7fcb109e13719d2ba"
server_key_version: "encryptionkey@0"
I20260806 04:51:52.554371 19191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:52.565748 19191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:52.566387 19191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:52.566663 19191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:52.583858 19191 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:37479
I20260806 04:51:52.583889 19258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:37479 every 8 connection(s)
I20260806 04:51:52.585084 19191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:51:52.588363 19259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:52.593750 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 19191
I20260806 04:51:52.593940 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:51:52.594523 13906 external_mini_cluster.cc:1428] Setting key 56c1c3f0c9caf81c5faa316480b5d616
I20260806 04:51:52.598132 19259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7: Bootstrap starting.
I20260806 04:51:52.601305 19259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:52.602461 19259 log.cc:824] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:52.605259 19259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7: No bootstrap required, opened a new log
I20260806 04:51:52.607303 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:52.596217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.13.148.190:37479)
0806 04:51:52.597288 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:52.597298 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:52.597337 (+ 39us) server_negotiation.cc:407] Connection header received
0806 04:51:52.598928 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:52.598949 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:52.599374 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:52.599677 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:52.600280 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:52.601652 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:52.603371 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:52.604018 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:52.604134 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:52.605326 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:52.605338 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:52.605355 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:52.605831 (+ 476us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:52.605908 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:52.605926 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:52.606002 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:52.606221 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 04:51:52.606501 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":366,"threads_started":1}
I20260806 04:51:52.610957 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:51:52.612720 19259 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce23ab0ba7164fcaa601e81cfb2db5d7" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37479 } }
I20260806 04:51:52.613101 19259 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:52.613237 19259 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce23ab0ba7164fcaa601e81cfb2db5d7, State: Initialized, Role: FOLLOWER
I20260806 04:51:52.613853 19259 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce23ab0ba7164fcaa601e81cfb2db5d7" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37479 } }
I20260806 04:51:52.614136 19259 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:51:52.614269 19259 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:51:52.614483 19259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:52.617619 19259 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce23ab0ba7164fcaa601e81cfb2db5d7" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37479 } }
I20260806 04:51:52.618201 19259 leader_election.cc:302] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce23ab0ba7164fcaa601e81cfb2db5d7; no voters:
I20260806 04:51:52.618881 19259 leader_election.cc:288] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:51:52.619045 19264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:52.619903 19264 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [term 1 LEADER]: Becoming Leader. State: Replica: ce23ab0ba7164fcaa601e81cfb2db5d7, State: Running, Role: LEADER
I20260806 04:51:52.620653 19264 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce23ab0ba7164fcaa601e81cfb2db5d7" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37479 } }
I20260806 04:51:52.623044 19259 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:51:52.623616 19266 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce23ab0ba7164fcaa601e81cfb2db5d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce23ab0ba7164fcaa601e81cfb2db5d7" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37479 } } }
I20260806 04:51:52.623970 19266 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:52.624473 19265 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce23ab0ba7164fcaa601e81cfb2db5d7. Latest consensus state: current_term: 1 leader_uuid: "ce23ab0ba7164fcaa601e81cfb2db5d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce23ab0ba7164fcaa601e81cfb2db5d7" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37479 } } }
I20260806 04:51:52.624732 19265 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:52.626912 19268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:51:52.633397 19268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:51:52.645768 19268 catalog_manager.cc:1347] Generated new cluster ID: f678fed1e5d24cd29ba015a6719fab0a
I20260806 04:51:52.645864 19268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:51:52.681962 19268 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:51:52.683626 19268 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:51:52.691185 19268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce23ab0ba7164fcaa601e81cfb2db5d7: Generated new TSK 0
I20260806 04:51:52.692094 19268 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 04:51:52.733800 19291 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 04:51:52.733992 19291 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 04:51:52.736163 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:52.707560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.13.148.190:37479)
0806 04:51:52.707719 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:52.707726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:52.708136 (+ 410us) server_negotiation.cc:407] Connection header received
0806 04:51:52.708431 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:52.708434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:52.708494 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:52.708633 (+ 139us) server_negotiation.cc:226] Negotiated authn=JWT
0806 04:51:52.709184 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:52.709902 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:52.711282 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:52.711708 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:52.711829 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:52.712292 (+ 463us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 04:51:52.735504 (+ 23212us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36071/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 04:51:52.735856 (+ 352us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36864: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36071/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 04:51:52.736284 19262 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36864: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36071/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 04:51:52.737834 19290 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:51:52.707507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:37479 (local address 127.0.0.1:36864)
0806 04:51:52.708042 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:52.708055 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:52.708230 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:52.708658 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:52.708663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:52.708675 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0806 04:51:52.709045 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:52.709053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:52.710048 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:52.710057 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:52.711127 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:52.711137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:52.711866 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:52.711872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:52.712000 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:52.712049 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 04:51:52.735766 (+ 23717us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36071/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 04:51:52.736076 (+ 310us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.148.190:37479: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36071/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":288,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.13.148.190:37479: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36071/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 04:51:52.739835 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 19191
I20260806 04:51:52.740983 19254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:52.826140 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 19191
2026-08-06T04:51:52Z chronyd exiting
I20260806 04:51:52.879729 13906 test_util.cc:175] -----------------------------------------------
I20260806 04:51:52.879840 13906 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991872418975-13906-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (717 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 04:51:52.883176 13906 mini_oidc.cc:152] Starting JWKS server
I20260806 04:51:52.884755 13906 webserver.cc:460] Webserver started at https://127.0.0.1:43337/ using document root <none> and password file <none>
I20260806 04:51:52.884997 13906 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 04:51:52.885496 13906 webserver.cc:460] Webserver started at http://127.0.0.1:34409/ using document root <none> and password file <none>
2026-08-06T04:51:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:51:52Z Disabled control of system clock
I20260806 04:51:52.911502 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:43983
--webserver_interface=127.13.148.190
--webserver_port=0
--builtin_ntp_servers=127.13.148.148:43815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:43983
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43337/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/testchainca.cert with env {}
W20260806 04:51:53.202739 19299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:53.202976 19299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:53.203068 19299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:53.209376 19299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:51:53.209473 19299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:53.209573 19299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:53.209682 19299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:51:53.209745 19299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 04:51:53.209849 19299 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 04:51:53.209905 19299 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43337/jwks.json
I20260806 04:51:53.218322 19299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:43815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:43983
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:43983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43337/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:53.219434 19299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:53.220670 19299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:53.231271 19307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:53.231240 19305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:53.231307 19304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:53.231874 19299 server_base.cc:1034] running on GCE node
I20260806 04:51:53.232640 19299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:53.233810 19299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:53.235018 19299 hybrid_clock.cc:648] HybridClock initialized: now 1785991913234994 us; error 35 us; skew 500 ppm
I20260806 04:51:53.235314 19299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:51:53.237442 19299 webserver.cc:460] Webserver started at http://127.13.148.190:43167/ using document root <none> and password file <none>
I20260806 04:51:53.238109 19299 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:53.238258 19299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:53.238616 19299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:53.241312 19299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "b4ab24dec16f4bf0b640bbee304621b4"
format_stamp: "Formatted at 2026-08-06 04:51:53 on dist-test-slave-1x4g"
server_key: "62cfbb2e4e973a8f62b36aa5895bb041"
server_key_iv: "00797390bd18bb6d55366526174ef85f"
server_key_version: "encryptionkey@0"
I20260806 04:51:53.242034 19299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "b4ab24dec16f4bf0b640bbee304621b4"
format_stamp: "Formatted at 2026-08-06 04:51:53 on dist-test-slave-1x4g"
server_key: "62cfbb2e4e973a8f62b36aa5895bb041"
server_key_iv: "00797390bd18bb6d55366526174ef85f"
server_key_version: "encryptionkey@0"
I20260806 04:51:53.246505 19299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:51:53.249513 19313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:53.250711 19299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:51:53.250882 19299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "b4ab24dec16f4bf0b640bbee304621b4"
format_stamp: "Formatted at 2026-08-06 04:51:53 on dist-test-slave-1x4g"
server_key: "62cfbb2e4e973a8f62b36aa5895bb041"
server_key_iv: "00797390bd18bb6d55366526174ef85f"
server_key_version: "encryptionkey@0"
I20260806 04:51:53.251075 19299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:53.268347 19299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:53.269183 19299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:53.269490 19299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:53.286463 19299 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:43983
I20260806 04:51:53.286471 19364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:43983 every 8 connection(s)
I20260806 04:51:53.287659 19299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:51:53.291117 19365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:53.297906 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 19299
I20260806 04:51:53.298115 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:51:53.298753 13906 external_mini_cluster.cc:1428] Setting key 48e5910464bd10a54899408fa3719a6b
I20260806 04:51:53.300228 19365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4: Bootstrap starting.
I20260806 04:51:53.303422 19365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:53.304373 19365 log.cc:824] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:53.306984 19365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4: No bootstrap required, opened a new log
I20260806 04:51:53.311939 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:53.300587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42916 (local address 127.13.148.190:43983)
0806 04:51:53.301479 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:53.301488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:53.301525 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:51:53.303139 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:53.303160 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:53.303662 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:53.304002 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:53.304557 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:53.305926 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:53.307174 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:53.308151 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:53.308330 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:53.309288 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:53.309298 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:53.309314 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:53.310055 (+ 741us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:53.310160 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:53.310184 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:53.310303 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:53.310483 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 04:51:53.310802 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":322,"threads_started":1}
I20260806 04:51:53.315411 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:51:53.316157 19365 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab24dec16f4bf0b640bbee304621b4" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 43983 } }
I20260806 04:51:53.316479 19365 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:53.316603 19365 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ab24dec16f4bf0b640bbee304621b4, State: Initialized, Role: FOLLOWER
I20260806 04:51:53.317175 19365 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab24dec16f4bf0b640bbee304621b4" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 43983 } }
I20260806 04:51:53.317400 19365 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:51:53.317584 19365 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:51:53.317754 19365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:53.320011 19365 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab24dec16f4bf0b640bbee304621b4" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 43983 } }
I20260806 04:51:53.320430 19365 leader_election.cc:302] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4ab24dec16f4bf0b640bbee304621b4; no voters:
I20260806 04:51:53.320966 19365 leader_election.cc:288] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:51:53.321115 19371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:53.321770 19371 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [term 1 LEADER]: Becoming Leader. State: Replica: b4ab24dec16f4bf0b640bbee304621b4, State: Running, Role: LEADER
I20260806 04:51:53.322270 19371 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab24dec16f4bf0b640bbee304621b4" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 43983 } }
I20260806 04:51:53.322880 19365 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:51:53.324908 19373 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4ab24dec16f4bf0b640bbee304621b4. Latest consensus state: current_term: 1 leader_uuid: "b4ab24dec16f4bf0b640bbee304621b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab24dec16f4bf0b640bbee304621b4" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 43983 } } }
I20260806 04:51:53.325261 19373 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:53.325081 19372 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4ab24dec16f4bf0b640bbee304621b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab24dec16f4bf0b640bbee304621b4" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 43983 } } }
I20260806 04:51:53.325428 19372 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:53.326889 19379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:51:53.331354 19379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:51:53.340052 19379 catalog_manager.cc:1347] Generated new cluster ID: 1aa3f20b6a034d8a8a3a476486632ae3
I20260806 04:51:53.340152 19379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:51:53.356323 19379 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:51:53.357550 19379 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:51:53.365643 19379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4ab24dec16f4bf0b640bbee304621b4: Generated new TSK 0
I20260806 04:51:53.366290 19379 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 04:51:53.399763 19397 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 04:51:53.399879 19397 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 04:51:53.401043 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:53.382626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.13.148.190:43983)
0806 04:51:53.382782 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:53.382787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:53.383094 (+ 307us) server_negotiation.cc:407] Connection header received
0806 04:51:53.383272 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:53.383277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:53.383367 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:53.383516 (+ 149us) server_negotiation.cc:226] Negotiated authn=JWT
0806 04:51:53.383888 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:53.384875 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:53.386025 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:53.386710 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:53.386834 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:53.387237 (+ 403us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 04:51:53.400393 (+ 13156us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 04:51:53.400766 (+ 373us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42924: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 04:51:53.401086 19396 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:51:53.382496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:43983 (local address 127.0.0.1:42924)
0806 04:51:53.383010 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:53.383022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:53.383121 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:53.383525 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:53.383527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:53.383535 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 04:51:53.383751 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:53.383757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:53.384978 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:53.384981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:53.385904 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:53.385913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:53.386855 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:53.386859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:53.386982 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:53.387010 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 04:51:53.400607 (+ 13597us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 04:51:53.400772 (+ 165us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.148.190:43983: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":281,"threads_started":1}
W20260806 04:51:53.401158 19368 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42924: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260806 04:51:53.402168 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 19299
I20260806 04:51:53.403395 19360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:53.487354 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 19299
2026-08-06T04:51:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (659 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 04:51:53.543262 13906 mini_oidc.cc:152] Starting JWKS server
I20260806 04:51:53.545183 13906 webserver.cc:460] Webserver started at https://127.0.0.1:46369/ using document root <none> and password file <none>
I20260806 04:51:53.545414 13906 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 04:51:53.546002 13906 webserver.cc:460] Webserver started at http://127.0.0.1:44317/ using document root <none> and password file <none>
2026-08-06T04:51:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:51:53Z Disabled control of system clock
I20260806 04:51:53.572182 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:34041
--webserver_interface=127.13.148.190
--webserver_port=0
--builtin_ntp_servers=127.13.148.148:41371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:34041
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46369/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 04:51:53.875901 19404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:53.876230 19404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:53.876390 19404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:53.883205 19404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:51:53.883369 19404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:53.883466 19404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:53.883570 19404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:51:53.883659 19404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 04:51:53.883749 19404 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 04:51:53.883841 19404 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46369/jwks.json
I20260806 04:51:53.892203 19404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:41371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:34041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:34041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46369/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:53.893471 19404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:53.895115 19404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:53.906486 19409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:53.906563 19410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:53.906507 19412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:53.907235 19404 server_base.cc:1034] running on GCE node
I20260806 04:51:53.907863 19404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:53.909097 19404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:53.910291 19404 hybrid_clock.cc:648] HybridClock initialized: now 1785991913910255 us; error 36 us; skew 500 ppm
I20260806 04:51:53.910595 19404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:51:53.912519 19404 webserver.cc:460] Webserver started at http://127.13.148.190:40593/ using document root <none> and password file <none>
I20260806 04:51:53.913211 19404 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:53.913290 19404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:53.913545 19404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:53.915949 19404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "bda222ba4dd548c39cab7ff44b0d40d0"
format_stamp: "Formatted at 2026-08-06 04:51:53 on dist-test-slave-1x4g"
server_key: "2457761a14dee4bed12f70cfebcf8f48"
server_key_iv: "77ebdb820c6b193a932acc0c8b1df487"
server_key_version: "encryptionkey@0"
I20260806 04:51:53.916671 19404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "bda222ba4dd548c39cab7ff44b0d40d0"
format_stamp: "Formatted at 2026-08-06 04:51:53 on dist-test-slave-1x4g"
server_key: "2457761a14dee4bed12f70cfebcf8f48"
server_key_iv: "77ebdb820c6b193a932acc0c8b1df487"
server_key_version: "encryptionkey@0"
I20260806 04:51:53.921290 19404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 04:51:53.924052 19418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:53.925247 19404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:51:53.925416 19404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "bda222ba4dd548c39cab7ff44b0d40d0"
format_stamp: "Formatted at 2026-08-06 04:51:53 on dist-test-slave-1x4g"
server_key: "2457761a14dee4bed12f70cfebcf8f48"
server_key_iv: "77ebdb820c6b193a932acc0c8b1df487"
server_key_version: "encryptionkey@0"
I20260806 04:51:53.925611 19404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:53.938339 19404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:53.939007 19404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:53.939276 19404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:53.955756 19404 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:34041
I20260806 04:51:53.955783 19469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:34041 every 8 connection(s)
I20260806 04:51:53.957046 19404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:51:53.957508 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 19404
I20260806 04:51:53.957703 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:51:53.958477 13906 external_mini_cluster.cc:1428] Setting key 0e7d5c303ef4ce94fb055ae5c1e5a562
I20260806 04:51:53.961009 19470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:53.969732 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:53.960180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.13.148.190:34041)
0806 04:51:53.961173 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:53.961182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:53.961218 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:51:53.962531 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:53.962550 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:53.962918 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:53.963265 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:53.963729 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:53.964920 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:53.966030 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:53.966575 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:53.966744 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:53.967794 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:53.967800 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:53.967810 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:53.968274 (+ 464us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:53.968336 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:53.968357 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:53.968488 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:53.968695 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 04:51:53.968956 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":377,"threads_started":1}
I20260806 04:51:53.970048 19470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0: Bootstrap starting.
I20260806 04:51:53.973171 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:51:53.973271 19470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:53.974157 19470 log.cc:824] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:53.976992 19470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0: No bootstrap required, opened a new log
I20260806 04:51:53.983832 19470 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda222ba4dd548c39cab7ff44b0d40d0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34041 } }
I20260806 04:51:53.984112 19470 raft_consensus.cc:374] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:53.984174 19470 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda222ba4dd548c39cab7ff44b0d40d0, State: Initialized, Role: FOLLOWER
I20260806 04:51:53.984596 19470 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda222ba4dd548c39cab7ff44b0d40d0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34041 } }
I20260806 04:51:53.984843 19470 raft_consensus.cc:388] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:51:53.984946 19470 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:51:53.985067 19470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:53.986860 19470 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda222ba4dd548c39cab7ff44b0d40d0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34041 } }
I20260806 04:51:53.987252 19470 leader_election.cc:302] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda222ba4dd548c39cab7ff44b0d40d0; no voters:
I20260806 04:51:53.987782 19470 leader_election.cc:288] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:51:53.987905 19475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:53.988508 19475 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [term 1 LEADER]: Becoming Leader. State: Replica: bda222ba4dd548c39cab7ff44b0d40d0, State: Running, Role: LEADER
I20260806 04:51:53.989104 19475 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda222ba4dd548c39cab7ff44b0d40d0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34041 } }
I20260806 04:51:53.989869 19470 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:51:53.991830 19477 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda222ba4dd548c39cab7ff44b0d40d0. Latest consensus state: current_term: 1 leader_uuid: "bda222ba4dd548c39cab7ff44b0d40d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda222ba4dd548c39cab7ff44b0d40d0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34041 } } }
I20260806 04:51:53.992282 19477 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:53.991910 19476 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bda222ba4dd548c39cab7ff44b0d40d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda222ba4dd548c39cab7ff44b0d40d0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34041 } } }
I20260806 04:51:53.992556 19476 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0 [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:53.994026 19484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:51:53.998840 19484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:51:54.006412 19484 catalog_manager.cc:1347] Generated new cluster ID: 7b9e43d7cb3347dd8c9952a8919dc138
I20260806 04:51:54.006512 19484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:51:54.014213 19484 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:51:54.015240 19484 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:51:54.025432 19484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bda222ba4dd548c39cab7ff44b0d40d0: Generated new TSK 0
I20260806 04:51:54.026126 19484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:51:54.053743 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:54.046191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.13.148.190:34041)
0806 04:51:54.046385 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:54.046391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:54.046788 (+ 397us) server_negotiation.cc:407] Connection header received
0806 04:51:54.046970 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:54.046975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:54.047057 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:54.047222 (+ 165us) server_negotiation.cc:226] Negotiated authn=JWT
0806 04:51:54.047601 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:54.048591 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:54.049926 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:54.050650 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:54.050784 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:54.051187 (+ 403us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 04:51:54.053184 (+ 1997us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46369/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 04:51:54.053459 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46369/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 04:51:54.053865 19473 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46369/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 04:51:54.055294 19500 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:51:54.046053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:34041 (local address 127.0.0.1:46084)
0806 04:51:54.046701 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:54.046715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:54.046818 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:54.047226 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:54.047229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:54.047238 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 04:51:54.047470 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:54.047477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:54.048709 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:54.048712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:54.049787 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:54.049799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:54.050823 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:54.050827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:54.050934 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:54.050968 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 04:51:54.053356 (+ 2388us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46369/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 04:51:54.054868 (+ 1512us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.148.190:34041: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46369/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":1265,"thread_start_us":378,"threads_started":1}
I20260806 04:51:54.056556 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 19404
I20260806 04:51:54.057694 19465 generic_service.cc:196] Checking for leaks (request via RPC)
E20260806 04:51:54.065052 19501 webserver.cc:533] Webserver: sslize failed:
E20260806 04:51:54.065124 19501 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 04:51:54.139750 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 19404
2026-08-06T04:51:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (651 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T04:51:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:51:54Z Disabled control of system clock
I20260806 04:51:54.219048 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:44567
--webserver_interface=127.13.148.190
--webserver_port=0
--builtin_ntp_servers=127.13.148.148:44947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:44567
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/testchainca.cert with env {}
W20260806 04:51:54.557886 19506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:54.558553 19506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:54.558930 19506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:54.575791 19506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:51:54.576040 19506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:54.576195 19506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:54.576496 19506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:51:54.576673 19506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 04:51:54.576954 19506 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 04:51:54.577116 19506 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 04:51:54.601373 19506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:44947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:44567
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:44567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:54.604465 19506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:54.607782 19506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:54.619398 19511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:54.619590 19512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:54.620867 19514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:54.621538 19506 server_base.cc:1034] running on GCE node
I20260806 04:51:54.622219 19506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:54.623442 19506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:54.624706 19506 hybrid_clock.cc:648] HybridClock initialized: now 1785991914624650 us; error 37 us; skew 500 ppm
I20260806 04:51:54.625068 19506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:51:54.627529 19506 webserver.cc:460] Webserver started at http://127.13.148.190:42507/ using document root <none> and password file <none>
I20260806 04:51:54.628170 19506 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:54.628295 19506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:54.628583 19506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:54.631412 19506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "870db88e24674904bf18f81c631d9dea"
format_stamp: "Formatted at 2026-08-06 04:51:54 on dist-test-slave-1x4g"
server_key: "aa66799f4df4eec57bd731d4f251f49d"
server_key_iv: "2e5707089b2cdece9abdc7ea76cfb8c3"
server_key_version: "encryptionkey@0"
I20260806 04:51:54.632193 19506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "870db88e24674904bf18f81c631d9dea"
format_stamp: "Formatted at 2026-08-06 04:51:54 on dist-test-slave-1x4g"
server_key: "aa66799f4df4eec57bd731d4f251f49d"
server_key_iv: "2e5707089b2cdece9abdc7ea76cfb8c3"
server_key_version: "encryptionkey@0"
I20260806 04:51:54.637081 19506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260806 04:51:54.640244 19520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:54.641440 19506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:51:54.641618 19506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "870db88e24674904bf18f81c631d9dea"
format_stamp: "Formatted at 2026-08-06 04:51:54 on dist-test-slave-1x4g"
server_key: "aa66799f4df4eec57bd731d4f251f49d"
server_key_iv: "2e5707089b2cdece9abdc7ea76cfb8c3"
server_key_version: "encryptionkey@0"
I20260806 04:51:54.641811 19506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:54.672122 19506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:54.672812 19506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:54.673064 19506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:54.689910 19506 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:44567
I20260806 04:51:54.689932 19571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:44567 every 8 connection(s)
I20260806 04:51:54.691118 19506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:51:54.694375 19572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:54.695891 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 19506
I20260806 04:51:54.696075 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:51:54.696693 13906 external_mini_cluster.cc:1428] Setting key 804c53b567dec4ef51fd1bfed87bdeb7
I20260806 04:51:54.705340 19572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea: Bootstrap starting.
I20260806 04:51:54.708751 19572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:54.709681 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:54.698463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.13.148.190:44567)
0806 04:51:54.699449 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:54.699458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:54.699514 (+ 56us) server_negotiation.cc:407] Connection header received
0806 04:51:54.701131 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:54.701150 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:54.701567 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:54.701878 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:54.702450 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:54.703733 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:54.705491 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:54.706140 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:54.706300 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:54.707580 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:54.707587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:54.707599 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:51:54.708067 (+ 468us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:54.708133 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:51:54.708152 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:54.708236 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:54.708458 (+ 222us) server_negotiation.cc:299] Negotiation successful
0806 04:51:54.708749 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":373,"threads_started":1}
I20260806 04:51:54.709797 19572 log.cc:824] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea: Log is configured to *not* fsync() on all Append() calls
I20260806 04:51:54.713044 19572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea: No bootstrap required, opened a new log
I20260806 04:51:54.714028 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:51:54.720459 19572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870db88e24674904bf18f81c631d9dea" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44567 } }
I20260806 04:51:54.720916 19572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:54.721031 19572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870db88e24674904bf18f81c631d9dea, State: Initialized, Role: FOLLOWER
I20260806 04:51:54.721532 19572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870db88e24674904bf18f81c631d9dea" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44567 } }
I20260806 04:51:54.721736 19572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:51:54.721880 19572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:51:54.722036 19572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:54.723845 19572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870db88e24674904bf18f81c631d9dea" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44567 } }
I20260806 04:51:54.724354 19572 leader_election.cc:302] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 870db88e24674904bf18f81c631d9dea; no voters:
I20260806 04:51:54.725018 19572 leader_election.cc:288] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:51:54.725152 19577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:54.726470 19577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [term 1 LEADER]: Becoming Leader. State: Replica: 870db88e24674904bf18f81c631d9dea, State: Running, Role: LEADER
I20260806 04:51:54.726975 19572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:51:54.726926 19577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870db88e24674904bf18f81c631d9dea" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44567 } }
I20260806 04:51:54.729723 19579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 870db88e24674904bf18f81c631d9dea. Latest consensus state: current_term: 1 leader_uuid: "870db88e24674904bf18f81c631d9dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870db88e24674904bf18f81c631d9dea" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44567 } } }
I20260806 04:51:54.730139 19579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:54.730502 19578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "870db88e24674904bf18f81c631d9dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870db88e24674904bf18f81c631d9dea" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44567 } } }
I20260806 04:51:54.730885 19578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:54.733637 19584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:51:54.737823 19584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:51:54.746037 19584 catalog_manager.cc:1347] Generated new cluster ID: 2381b2833c9045eb82edefe02c00228f
I20260806 04:51:54.746135 19584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:51:54.776854 19584 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:51:54.778069 19584 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:51:54.788825 19584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 870db88e24674904bf18f81c631d9dea: Generated new TSK 0
I20260806 04:51:54.789633 19584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:51:54.913481 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:54.808902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56926 (local address 127.13.148.190:44567)
0806 04:51:54.809113 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:54.809121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:54.809427 (+ 306us) server_negotiation.cc:407] Connection header received
0806 04:51:54.809687 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:54.809690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:54.809752 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:54.809877 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0806 04:51:54.810394 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:54.811114 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:54.813060 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:54.813466 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:54.813548 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:54.814126 (+ 578us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 04:51:54.912864 (+ 98738us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 04:51:54.913211 (+ 347us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56926: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 04:51:54.913666 19575 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56926: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 04:51:54.913952 19601 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:51:54.808761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:44567 (local address 127.0.0.1:56926)
0806 04:51:54.809328 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:54.809345 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:54.809518 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:54.809913 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:54.809921 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:54.809934 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0806 04:51:54.810243 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:54.810256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:54.811245 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:54.811251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:54.812932 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:54.812946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:54.813621 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:54.813627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:54.813801 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:54.813842 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 04:51:54.913167 (+ 99325us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 04:51:54.913515 (+ 348us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.148.190:44567: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":257,"threads_started":1}
I20260806 04:51:54.915272 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 19506
I20260806 04:51:54.916517 19567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:55.010736 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 19506
2026-08-06T04:51:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (852 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (381 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (374 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:51:55 dist-test-slave-1x4g krb5kdc[19616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:51:55 dist-test-slave-1x4g krb5kdc[19616](info): set up 2 sockets
Aug 06 04:51:55 dist-test-slave-1x4g krb5kdc[19616](info): commencing operation
krb5kdc: starting...
W20260806 04:51:56.777487 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.011s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:51:56 dist-test-slave-1x4g krb5kdc[19616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:51:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:51:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:51:56.970072 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:45361
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:38357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:45361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:57.263178 19630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:57.263474 19630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:57.263634 19630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:57.269744 19630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:51:57.269886 19630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:57.269985 19630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:57.270087 19630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:51:57.270172 19630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:51:57.279080 19630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:38357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:45361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:45361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:57.280308 19630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:57.281533 19630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:57.292182 19635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:57.292114 19638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:57.292152 19636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:57.293078 19630 server_base.cc:1034] running on GCE node
I20260806 04:51:57.293731 19630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:57.294869 19630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:57.296082 19630 hybrid_clock.cc:648] HybridClock initialized: now 1785991917296044 us; error 46 us; skew 500 ppm
I20260806 04:51:57.296382 19630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:57 dist-test-slave-1x4g krb5kdc[19616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991917, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:57.302809 19630 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:51:57.304100 19630 webserver.cc:460] Webserver started at http://127.13.148.190:35329/ using document root <none> and password file <none>
I20260806 04:51:57.304720 19630 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:57.304883 19630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:57.305176 19630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:57.307717 19630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "ec2783f2bf4b4fdeaf71edc711d5828d"
format_stamp: "Formatted at 2026-08-06 04:51:57 on dist-test-slave-1x4g"
server_key: "f7fde1426a9f438866a31df70fdfdaee"
server_key_iv: "632580c72dd79ba57331b271b8062a34"
server_key_version: "encryptionkey@0"
I20260806 04:51:57.308424 19630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "ec2783f2bf4b4fdeaf71edc711d5828d"
format_stamp: "Formatted at 2026-08-06 04:51:57 on dist-test-slave-1x4g"
server_key: "f7fde1426a9f438866a31df70fdfdaee"
server_key_iv: "632580c72dd79ba57331b271b8062a34"
server_key_version: "encryptionkey@0"
I20260806 04:51:57.313345 19630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 04:51:57.316363 19646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:57.317633 19630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:51:57.317811 19630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "ec2783f2bf4b4fdeaf71edc711d5828d"
format_stamp: "Formatted at 2026-08-06 04:51:57 on dist-test-slave-1x4g"
server_key: "f7fde1426a9f438866a31df70fdfdaee"
server_key_iv: "632580c72dd79ba57331b271b8062a34"
server_key_version: "encryptionkey@0"
I20260806 04:51:57.318019 19630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:57.336664 19630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:57.340234 19630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:57.340514 19630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:57.358004 19630 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:45361
I20260806 04:51:57.358042 19700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:45361 every 8 connection(s)
I20260806 04:51:57.359166 19630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:51:57.362491 19701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:51:57.365657 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 19630
I20260806 04:51:57.365880 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:51:57.366595 13906 external_mini_cluster.cc:1428] Setting key ddd7cb6840b569a24c8937dd25f5f0c4
I20260806 04:51:57.374466 19701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d: Bootstrap starting.
I20260806 04:51:57.378058 19701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d: Neither blocks nor log segments found. Creating new log.
I20260806 04:51:57.379362 19701 log.cc:824] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:51:57 dist-test-slave-1x4g krb5kdc[19616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:57.382193 19701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d: No bootstrap required, opened a new log
I20260806 04:51:57.388329 19704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:57.368438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33212 (local address 127.13.148.190:45361)
0806 04:51:57.369135 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:57.369145 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:57.369181 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:51:57.370728 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:57.370750 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:57.371135 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:57.371421 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:57.373049 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:57.374257 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:57.375795 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:57.376415 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:57.376540 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:57.381494 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:57.381502 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:57.381515 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:57.381548 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:57.384403 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:57.385130 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:57.385146 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:57.385151 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:57.385312 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:57.385713 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:57.385734 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:57.385736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:57.386249 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:57.386569 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:57.386991 (+ 422us) server_negotiation.cc:299] Negotiation successful
0806 04:51:57.387383 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":343,"threads_started":1}
I20260806 04:51:57.389508 19701 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2783f2bf4b4fdeaf71edc711d5828d" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45361 } }
I20260806 04:51:57.389976 19701 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:51:57.390093 19701 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2783f2bf4b4fdeaf71edc711d5828d, State: Initialized, Role: FOLLOWER
I20260806 04:51:57.390640 19701 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2783f2bf4b4fdeaf71edc711d5828d" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45361 } }
I20260806 04:51:57.390915 19701 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:51:57.391054 19701 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:51:57.391213 19701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:51:57.393241 19701 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2783f2bf4b4fdeaf71edc711d5828d" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45361 } }
I20260806 04:51:57.393714 19701 leader_election.cc:302] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec2783f2bf4b4fdeaf71edc711d5828d; no voters:
I20260806 04:51:57.394256 19701 leader_election.cc:288] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:51:57.394376 19706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:51:57.394981 19706 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [term 1 LEADER]: Becoming Leader. State: Replica: ec2783f2bf4b4fdeaf71edc711d5828d, State: Running, Role: LEADER
I20260806 04:51:57.395440 19706 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2783f2bf4b4fdeaf71edc711d5828d" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45361 } }
I20260806 04:51:57.396029 19701 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:51:57.398160 19707 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec2783f2bf4b4fdeaf71edc711d5828d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2783f2bf4b4fdeaf71edc711d5828d" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45361 } } }
I20260806 04:51:57.398419 19708 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec2783f2bf4b4fdeaf71edc711d5828d. Latest consensus state: current_term: 1 leader_uuid: "ec2783f2bf4b4fdeaf71edc711d5828d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2783f2bf4b4fdeaf71edc711d5828d" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45361 } } }
I20260806 04:51:57.399096 19707 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:57.399186 19708 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d [sys.catalog]: This master's current role is: LEADER
I20260806 04:51:57.400552 19712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:51:57.404904 19712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:51:57.413555 19712 catalog_manager.cc:1347] Generated new cluster ID: 8ea599e052004c428bec0305ef3a80df
I20260806 04:51:57.413653 19712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:51:57.446089 19712 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:51:57.447742 19712 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:51:57.457692 19712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec2783f2bf4b4fdeaf71edc711d5828d: Generated new TSK 0
I20260806 04:51:57.458467 19712 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:51:57.512410 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:45361
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:38357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:57.809038 19729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:57.809306 19729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:57.809463 19729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:57.815822 19729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:57.815963 19729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:57.816109 19729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:51:57.824910 19729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:38357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:45361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:57.825974 19729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:57.827203 19729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:57.839298 19737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:57.839380 19735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:57.839401 19734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:57.840046 19729 server_base.cc:1034] running on GCE node
I20260806 04:51:57.840595 19729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:57.841463 19729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:57.842681 19729 hybrid_clock.cc:648] HybridClock initialized: now 1785991917842640 us; error 49 us; skew 500 ppm
I20260806 04:51:57.842968 19729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:57 dist-test-slave-1x4g krb5kdc[19616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991917, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:57.845757 19729 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:51:57.847028 19729 webserver.cc:460] Webserver started at http://127.13.148.129:37971/ using document root <none> and password file <none>
I20260806 04:51:57.847605 19729 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:57.847735 19729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:57.848006 19729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:57.850730 19729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "f02c957bcdb6434a9fa52e4b75f17aca"
format_stamp: "Formatted at 2026-08-06 04:51:57 on dist-test-slave-1x4g"
server_key: "09b11f51fe4cf558d548a75b95c6d535"
server_key_iv: "8db91e46ec030bf657d29e844391470b"
server_key_version: "encryptionkey@0"
I20260806 04:51:57.851440 19729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "f02c957bcdb6434a9fa52e4b75f17aca"
format_stamp: "Formatted at 2026-08-06 04:51:57 on dist-test-slave-1x4g"
server_key: "09b11f51fe4cf558d548a75b95c6d535"
server_key_iv: "8db91e46ec030bf657d29e844391470b"
server_key_version: "encryptionkey@0"
I20260806 04:51:57.856066 19729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260806 04:51:57.859071 19745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:57.860216 19729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:51:57.860394 19729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "f02c957bcdb6434a9fa52e4b75f17aca"
format_stamp: "Formatted at 2026-08-06 04:51:57 on dist-test-slave-1x4g"
server_key: "09b11f51fe4cf558d548a75b95c6d535"
server_key_iv: "8db91e46ec030bf657d29e844391470b"
server_key_version: "encryptionkey@0"
I20260806 04:51:57.860603 19729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:57.873229 19729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:57.876317 19729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:57.876575 19729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:57.877493 19729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:57.878649 19729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:57.878744 19729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:57.878871 19729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:57.878948 19729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:57.909708 19729 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:37445
I20260806 04:51:57.909783 19860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:37445 every 8 connection(s)
I20260806 04:51:57.910979 19729 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:51:57.920763 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 19729
I20260806 04:51:57.921068 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:51:57.921739 13906 external_mini_cluster.cc:1428] Setting key 239b357bd466df72ff628d71bfecff1f
Aug 06 04:51:57 dist-test-slave-1x4g krb5kdc[19616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991917, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:57.928974 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:57.913230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:44305 (local address 127.13.148.190:45361)
0806 04:51:57.913625 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:57.913629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:57.914539 (+ 910us) server_negotiation.cc:407] Connection header received
0806 04:51:57.916171 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:57.916176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:57.916260 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:57.916365 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:57.917846 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:57.918547 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:57.919957 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:57.920360 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:57.920499 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:57.924181 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:57.924187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:57.924191 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:57.924225 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:57.926393 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:57.927009 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:57.927017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:57.927019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:57.927126 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:57.927501 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:57.927506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:57.927510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:57.927743 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:57.927883 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:57.928425 (+ 542us) server_negotiation.cc:299] Negotiation successful
0806 04:51:57.928628 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":204,"threads_started":1}
I20260806 04:51:57.929757 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:57.913739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:45361 (local address 127.13.148.129:44305)
0806 04:51:57.914398 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:57.914434 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:57.915903 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:57.916611 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:57.916618 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:57.917164 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:57.917669 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:57.917684 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:57.918706 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:57.918709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:57.919807 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:57.919818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:57.920582 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:57.920591 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:57.920937 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:57.921610 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:57.921630 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:57.924000 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:57.926526 (+ 2526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:57.926533 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:57.926545 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:57.926886 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:57.927224 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:57.927228 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:57.927231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:57.927391 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:57.927855 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:57.927862 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:57.928136 (+ 274us) client_negotiation.cc:769] Sending connection context
0806 04:51:57.928359 (+ 223us) client_negotiation.cc:240] Negotiation successful
0806 04:51:57.928758 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":258,"threads_started":1}
I20260806 04:51:57.931263 19861 heartbeater.cc:347] Connected to a master server at 127.13.148.190:45361
I20260806 04:51:57.931524 19861 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:57.932137 19861 heartbeater.cc:511] Master 127.13.148.190:45361 requested a full tablet report, sending...
I20260806 04:51:57.933984 19665 ts_manager.cc:194] Registered new tserver with Master: f02c957bcdb6434a9fa52e4b75f17aca (127.13.148.129:37445)
I20260806 04:51:57.935271 19665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:44305
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:51:57.963981 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:45361
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:38357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:58.254382 19868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:58.254655 19868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:58.254750 19868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:58.261487 19868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:58.261574 19868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:58.261688 19868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:51:58.270141 19868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:38357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:45361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:58.271260 19868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:58.272403 19868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:58.283828 19876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:58.283890 19873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:58.283924 19874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:58.284657 19868 server_base.cc:1034] running on GCE node
I20260806 04:51:58.285166 19868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:58.285974 19868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:58.287181 19868 hybrid_clock.cc:648] HybridClock initialized: now 1785991918287152 us; error 39 us; skew 500 ppm
I20260806 04:51:58.287467 19868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:58 dist-test-slave-1x4g krb5kdc[19616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991918, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:58.290511 19868 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:51:58.291924 19868 webserver.cc:460] Webserver started at http://127.13.148.130:33373/ using document root <none> and password file <none>
I20260806 04:51:58.292551 19868 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:58.292697 19868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:58.293059 19868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:58.295570 19868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "98f3de76122447d289676949773af1f6"
format_stamp: "Formatted at 2026-08-06 04:51:58 on dist-test-slave-1x4g"
server_key: "8e258f8059ce25204ed7d08b51632ccc"
server_key_iv: "cf88cc77ef1edcf3c970cee56b6ba25d"
server_key_version: "encryptionkey@0"
I20260806 04:51:58.296345 19868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "98f3de76122447d289676949773af1f6"
format_stamp: "Formatted at 2026-08-06 04:51:58 on dist-test-slave-1x4g"
server_key: "8e258f8059ce25204ed7d08b51632ccc"
server_key_iv: "cf88cc77ef1edcf3c970cee56b6ba25d"
server_key_version: "encryptionkey@0"
I20260806 04:51:58.301173 19868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260806 04:51:58.304011 19883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:58.305189 19868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:51:58.305341 19868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "98f3de76122447d289676949773af1f6"
format_stamp: "Formatted at 2026-08-06 04:51:58 on dist-test-slave-1x4g"
server_key: "8e258f8059ce25204ed7d08b51632ccc"
server_key_iv: "cf88cc77ef1edcf3c970cee56b6ba25d"
server_key_version: "encryptionkey@0"
I20260806 04:51:58.305516 19868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:58.316989 19868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:58.320500 19868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:58.320816 19868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:58.321724 19868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:58.322908 19868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:58.323004 19868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:58.323158 19868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:58.323225 19868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:58.354277 19868 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:35911
I20260806 04:51:58.354352 19998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:35911 every 8 connection(s)
I20260806 04:51:58.355471 19868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:51:58.358984 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 19868
I20260806 04:51:58.359184 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:51:58.359704 13906 external_mini_cluster.cc:1428] Setting key a40fa5aa73e40f0a64fdfaa17b4906e6
Aug 06 04:51:58 dist-test-slave-1x4g krb5kdc[19616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991918, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:58.378351 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:58.357981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:43545 (local address 127.13.148.190:45361)
0806 04:51:58.358136 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:58.358140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:58.359539 (+ 1399us) server_negotiation.cc:407] Connection header received
0806 04:51:58.361752 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:58.361758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:58.361838 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:58.361949 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:58.363783 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:58.364791 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:58.366896 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:58.367541 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:58.367663 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:58.371820 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:58.371827 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:58.371831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:58.371866 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:58.375004 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:58.375939 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:58.375943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:58.375945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:58.376074 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:58.376536 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:58.376539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:58.376541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:58.376898 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:58.377014 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:58.377894 (+ 880us) server_negotiation.cc:299] Negotiation successful
0806 04:51:58.378049 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:51:58.379704 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:58.358526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:45361 (local address 127.13.148.130:43545)
0806 04:51:58.359365 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:58.359425 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:58.361480 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:58.362166 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:58.362176 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:58.362867 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:58.363604 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:58.363630 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:58.364962 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:58.364968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:58.366746 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:58.366760 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:58.367763 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:58.367769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:58.368035 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:58.368702 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:58.368727 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:58.371607 (+ 2880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:58.375130 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:58.375140 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:58.375155 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:58.375818 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:58.376184 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:58.376192 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:58.376198 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:58.376432 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:58.377044 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:58.377054 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:58.377485 (+ 431us) client_negotiation.cc:769] Sending connection context
0806 04:51:58.377807 (+ 322us) client_negotiation.cc:240] Negotiation successful
0806 04:51:58.378344 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":315,"threads_started":1}
I20260806 04:51:58.381163 19999 heartbeater.cc:347] Connected to a master server at 127.13.148.190:45361
I20260806 04:51:58.381440 19999 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:58.382072 19999 heartbeater.cc:511] Master 127.13.148.190:45361 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
I20260806 04:51:58.383464 19665 ts_manager.cc:194] Registered new tserver with Master: 98f3de76122447d289676949773af1f6 (127.13.148.130:35911)
I20260806 04:51:58.384231 19665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:43545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
I20260806 04:51:58.406970 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:45361
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:38357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:51:58.703820 20005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:51:58.704071 20005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:51:58.704238 20005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:51:58.711063 20005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:51:58.711153 20005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:51:58.711273 20005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:51:58.719966 20005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:38357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:45361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:51:58.721410 20005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:51:58.722770 20005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:51:58.734690 20013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:58.734658 20011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:51:58.734856 20010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:51:58.735577 20005 server_base.cc:1034] running on GCE node
I20260806 04:51:58.736191 20005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:51:58.737159 20005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:51:58.738391 20005 hybrid_clock.cc:648] HybridClock initialized: now 1785991918738352 us; error 53 us; skew 500 ppm
I20260806 04:51:58.738682 20005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:51:58 dist-test-slave-1x4g krb5kdc[19616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991918, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:58.741815 20005 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:51:58.743140 20005 webserver.cc:460] Webserver started at http://127.13.148.131:40609/ using document root <none> and password file <none>
I20260806 04:51:58.743737 20005 fs_manager.cc:359] Metadata directory not provided
I20260806 04:51:58.743829 20005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:51:58.744069 20005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:51:58.746735 20005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "87f57f0c8ad84ab9b803ae75cce6ce07"
format_stamp: "Formatted at 2026-08-06 04:51:58 on dist-test-slave-1x4g"
server_key: "3841675dd6bdfcae403866954a601c95"
server_key_iv: "b05c99d563c83a78755e9f7f2332ee90"
server_key_version: "encryptionkey@0"
I20260806 04:51:58.747457 20005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "87f57f0c8ad84ab9b803ae75cce6ce07"
format_stamp: "Formatted at 2026-08-06 04:51:58 on dist-test-slave-1x4g"
server_key: "3841675dd6bdfcae403866954a601c95"
server_key_iv: "b05c99d563c83a78755e9f7f2332ee90"
server_key_version: "encryptionkey@0"
I20260806 04:51:58.752301 20005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260806 04:51:58.755388 20020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:58.756904 20005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:51:58.757084 20005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "87f57f0c8ad84ab9b803ae75cce6ce07"
format_stamp: "Formatted at 2026-08-06 04:51:58 on dist-test-slave-1x4g"
server_key: "3841675dd6bdfcae403866954a601c95"
server_key_iv: "b05c99d563c83a78755e9f7f2332ee90"
server_key_version: "encryptionkey@0"
I20260806 04:51:58.757325 20005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:51:58.775198 20005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:51:58.779027 20005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:51:58.779228 20005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:51:58.780035 20005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:51:58.781278 20005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:51:58.781343 20005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:58.781426 20005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:51:58.781464 20005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:51:58.812759 20005 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:45819
I20260806 04:51:58.812886 20136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:45819 every 8 connection(s)
I20260806 04:51:58.814045 20005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:51:58.823876 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 20005
I20260806 04:51:58.824064 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:51:58.824587 13906 external_mini_cluster.cc:1428] Setting key 126b4d77fc97d6846a124cbf604a36bf
Aug 06 04:51:58 dist-test-slave-1x4g krb5kdc[19616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991918, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:51:58.835450 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:58.816434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:56267 (local address 127.13.148.190:45361)
0806 04:51:58.816653 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:58.816658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:58.817956 (+ 1298us) server_negotiation.cc:407] Connection header received
0806 04:51:58.819585 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:58.819590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:58.819653 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:58.819742 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:58.821292 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:58.822059 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:58.823624 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:58.824041 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:58.824160 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:58.828089 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:58.828097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:58.828103 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:58.828142 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:58.831876 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:58.832834 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:58.832840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:58.832845 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:58.833015 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:58.833429 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:58.833434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:58.833438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:58.833817 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:58.833983 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:58.834888 (+ 905us) server_negotiation.cc:299] Negotiation successful
0806 04:51:58.835100 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 04:51:58.836773 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:58.817081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:45361 (local address 127.13.148.131:56267)
0806 04:51:58.817791 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0806 04:51:58.817827 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 04:51:58.819325 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:51:58.819962 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:51:58.819970 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:51:58.820479 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:51:58.821132 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:58.821151 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:58.822200 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:58.822204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:58.823471 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:51:58.823482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:51:58.824283 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:51:58.824291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:51:58.824545 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:58.825234 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:51:58.825258 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:51:58.827889 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:51:58.832022 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:58.832030 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:58.832044 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:58.832667 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:58.833124 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:51:58.833129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:51:58.833133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:51:58.833313 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:51:58.833977 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:51:58.833984 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:51:58.834441 (+ 457us) client_negotiation.cc:769] Sending connection context
0806 04:51:58.834770 (+ 329us) client_negotiation.cc:240] Negotiation successful
0806 04:51:58.835650 (+ 880us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":277,"threads_started":1}
I20260806 04:51:58.838176 20137 heartbeater.cc:347] Connected to a master server at 127.13.148.190:45361
I20260806 04:51:58.838551 20137 heartbeater.cc:464] Registering TS with master...
I20260806 04:51:58.839481 20137 heartbeater.cc:511] Master 127.13.148.190:45361 requested a full tablet report, sending...
I20260806 04:51:58.841012 19665 ts_manager.cc:194] Registered new tserver with Master: 87f57f0c8ad84ab9b803ae75cce6ce07 (127.13.148.131:45819)
I20260806 04:51:58.841768 19665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:56267
I20260806 04:51:58.851585 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/test-admin.keytab.
Aug 06 04:51:58 dist-test-slave-1x4g krb5kdc[19616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:58.883543 13906 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 04:51:58.888206 20144 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:51:58.885910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50884 (local address 127.13.148.129:37445)
0806 04:51:58.886460 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:58.886470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:58.886500 (+ 30us) server_negotiation.cc:407] Connection header received
0806 04:51:58.886589 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:58.886595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:58.886796 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:58.886968 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:58.887807 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50884: BlockingRecv error: recv got EOF from 127.0.0.1:50884 (error 108)
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":296,"threads_started":1}
W20260806 04:51:58.888366 13906 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/krb5cc' of type FILE
Aug 06 04:51:58 dist-test-slave-1x4g krb5kdc[19616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:58.890502 13906 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 04:51:58 dist-test-slave-1x4g krb5kdc[19616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:58.891988 13906 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 04:51:58 dist-test-slave-1x4g krb5kdc[19616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:51:58.908038 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:58.893671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50888 (local address 127.13.148.129:37445)
0806 04:51:58.893868 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:58.893875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:58.894357 (+ 482us) server_negotiation.cc:407] Connection header received
0806 04:51:58.894509 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:58.894512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:58.894566 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:58.894637 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:58.895679 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:58.896587 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:58.897937 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:58.898436 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:58.898578 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:58.902638 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:58.902648 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:58.902660 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:58.902700 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:58.905374 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:58.906204 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:58.906210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:58.906212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:58.906328 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:58.906711 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:58.906714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:58.906716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:58.907022 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:58.907216 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:58.907593 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 04:51:58.907766 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 04:51:58.920356 20144 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:51:58.918594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50902 (local address 127.13.148.129:37445)
0806 04:51:58.918810 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:58.918817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:58.918969 (+ 152us) server_negotiation.cc:407] Connection header received
0806 04:51:58.919076 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:58.919083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:58.919190 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:58.919328 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:58.920073 (+ 745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50902: BlockingRecv error: recv got EOF from 127.0.0.1:50902 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 04:51:58.920382 13906 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/krb5cc' of type FILE
Aug 06 04:51:58 dist-test-slave-1x4g krb5kdc[19616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:58.922365 13906 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 04:51:58 dist-test-slave-1x4g krb5kdc[19616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:58.923689 13906 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 04:51:58 dist-test-slave-1x4g krb5kdc[19616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:51:58.937808 19861 heartbeater.cc:503] Master 127.13.148.190:45361 was elected leader, sending a full tablet report...
I20260806 04:51:58.938331 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:58.924856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.13.148.129:37445)
0806 04:51:58.925071 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:58.925080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:58.925426 (+ 346us) server_negotiation.cc:407] Connection header received
0806 04:51:58.925561 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:58.925565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:58.925609 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:58.925678 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:58.926865 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:58.927474 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:58.928601 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:58.929162 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:58.929315 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:58.933120 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:58.933125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:58.933128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:58.933155 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:58.935444 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:58.936021 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:58.936027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:58.936030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:58.936147 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:58.936503 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:58.936508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:58.936511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:58.936849 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:58.937101 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:58.937379 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 04:51:58.937568 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260806 04:51:58.942953 20144 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:51:58.941231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50916 (local address 127.13.148.129:37445)
0806 04:51:58.941413 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:58.941421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:58.941749 (+ 328us) server_negotiation.cc:407] Connection header received
0806 04:51:58.941880 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:58.941883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:58.941944 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:58.942010 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:58.942733 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50916: BlockingRecv error: recv got EOF from 127.0.0.1:50916 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 04:51:58.943157 13906 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991872418975-13906-0/krb5kdc/krb5cc' of type FILE
Aug 06 04:51:58 dist-test-slave-1x4g krb5kdc[19616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:58.944906 13906 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 04:51:58 dist-test-slave-1x4g krb5kdc[19616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:51:58.946055 13906 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 04:51:58 dist-test-slave-1x4g krb5kdc[19616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:51:58.961083 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:58.947247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50926 (local address 127.13.148.129:37445)
0806 04:51:58.947407 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:58.947412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:58.947619 (+ 207us) server_negotiation.cc:407] Connection header received
0806 04:51:58.947705 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:58.947709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:58.947750 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:58.947813 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:58.949062 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:58.949661 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:58.951710 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:58.952041 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:58.952118 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:58.956155 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:58.956160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:58.956163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:58.956191 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:58.958617 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:58.959164 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:58.959170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:58.959173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:58.959308 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:58.959638 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:58.959643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:58.959646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:58.959936 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:58.960260 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:58.960539 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 04:51:58.960766 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:51:58.962668 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 19729
I20260806 04:51:58.975332 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:58.963292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50934 (local address 127.13.148.129:37445)
0806 04:51:58.963460 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:58.963467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:58.963741 (+ 274us) server_negotiation.cc:407] Connection header received
0806 04:51:58.963847 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:58.963853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:58.963932 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:58.964029 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:58.964836 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:58.965897 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:58.966860 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:58.967588 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:58.967721 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:58.969459 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:58.969466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:58.969473 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:58.969511 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:58.972955 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:58.973501 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:58.973507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:58.973512 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:58.973673 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:58.973982 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:58.973988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:58.973993 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:58.974300 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:58.974549 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:58.974711 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 04:51:58.974919 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:51:58.975951 19856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:59.054095 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 19729
I20260806 04:51:59.080341 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 19868
Aug 06 04:51:59 dist-test-slave-1x4g krb5kdc[19616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:51:59.096262 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:59.081434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59844 (local address 127.13.148.130:35911)
0806 04:51:59.082082 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:59.082089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:59.082112 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:51:59.082177 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:59.082183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:59.082385 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:59.082577 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:59.083591 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:59.084512 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:59.085867 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:59.086358 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:59.086517 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:59.089985 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:59.089997 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:59.090011 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:59.090064 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:59.093183 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:59.094060 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:59.094066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:59.094068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:59.094185 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:59.094592 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:59.094595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:59.094596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:59.094894 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:59.095112 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:59.095562 (+ 450us) server_negotiation.cc:299] Negotiation successful
0806 04:51:59.095866 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":298,"threads_started":1}
I20260806 04:51:59.097787 19994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:59.172762 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 19868
I20260806 04:51:59.189280 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 20005
Aug 06 04:51:59 dist-test-slave-1x4g krb5kdc[19616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
I20260806 04:51:59.204643 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:51:59.190112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.13.148.131:45819)
0806 04:51:59.190673 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0806 04:51:59.190679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:51:59.190708 (+ 29us) server_negotiation.cc:407] Connection header received
0806 04:51:59.190768 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:51:59.190774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:51:59.190984 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:51:59.191159 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:51:59.192212 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:59.193390 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:59.194605 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:51:59.195117 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:51:59.195254 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:51:59.199231 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:51:59.199241 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:51:59.199253 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:51:59.199281 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:51:59.201922 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:59.202570 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:59.202578 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:59.202585 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:59.202697 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:51:59.203082 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:51:59.203087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:51:59.203090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:51:59.203493 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:51:59.203799 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 04:51:59.204096 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 04:51:59.204343 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":254,"threads_started":1}
I20260806 04:51:59.206104 20132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:59.281991 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 20005
I20260806 04:51:59.298696 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 19630
I20260806 04:51:59.299786 19696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:51:59.395221 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 19630
2026-08-06T04:51:59Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3629 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:51:59 dist-test-slave-1x4g krb5kdc[20165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:51:59 dist-test-slave-1x4g krb5kdc[20165](info): set up 2 sockets
Aug 06 04:51:59 dist-test-slave-1x4g krb5kdc[20165](info): commencing operation
krb5kdc: starting...
W20260806 04:52:00.422353 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:52:00 dist-test-slave-1x4g krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:52:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:52:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.190.keytab.
Entry for principal oryx/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.190.keytab.
I20260806 04:52:00.617908 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:38833
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.190.keytab
--principal=oryx/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:36971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:38833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:00.926337 20178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:00.926621 20178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:00.926708 20178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:00.933146 20178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:52:00.933238 20178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:00.933279 20178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:00.933331 20178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:52:00.933369 20178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:52:00.942417 20178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:38833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.190.keytab
--principal=oryx/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:38833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:00.943665 20178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:00.944872 20178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:00.955924 20183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:00.956007 20186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:00.955983 20184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:00.956810 20178 server_base.cc:1034] running on GCE node
I20260806 04:52:00.957499 20178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:00.958737 20178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:00.959969 20178 hybrid_clock.cc:648] HybridClock initialized: now 1785991920959894 us; error 33 us; skew 500 ppm
I20260806 04:52:00.960243 20178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:00 dist-test-slave-1x4g krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991920, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:00.963230 20178 init.cc:377] Logged in from keytab as oryx/127.13.148.190@KRBTEST.COM (short username oryx)
I20260806 04:52:00.964617 20178 webserver.cc:460] Webserver started at http://127.13.148.190:45147/ using document root <none> and password file <none>
I20260806 04:52:00.965279 20178 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:00.965381 20178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:00.965644 20178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:00.968300 20178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "8545f2e2779a47e3ac5cb4398a291b1b"
format_stamp: "Formatted at 2026-08-06 04:52:00 on dist-test-slave-1x4g"
server_key: "743f1bf948d14aebc3870d73fcf1faa9"
server_key_iv: "27530a37f7d79a5b18164bb52e960c9f"
server_key_version: "encryptionkey@0"
I20260806 04:52:00.969079 20178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "8545f2e2779a47e3ac5cb4398a291b1b"
format_stamp: "Formatted at 2026-08-06 04:52:00 on dist-test-slave-1x4g"
server_key: "743f1bf948d14aebc3870d73fcf1faa9"
server_key_iv: "27530a37f7d79a5b18164bb52e960c9f"
server_key_version: "encryptionkey@0"
I20260806 04:52:00.973840 20178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:52:00.977020 20193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:00.978245 20178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:52:00.978406 20178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "8545f2e2779a47e3ac5cb4398a291b1b"
format_stamp: "Formatted at 2026-08-06 04:52:00 on dist-test-slave-1x4g"
server_key: "743f1bf948d14aebc3870d73fcf1faa9"
server_key_iv: "27530a37f7d79a5b18164bb52e960c9f"
server_key_version: "encryptionkey@0"
I20260806 04:52:00.978595 20178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:01.022536 20178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:01.026749 20178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:01.027035 20178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:01.045344 20178 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:38833
I20260806 04:52:01.045428 20244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:38833 every 8 connection(s)
I20260806 04:52:01.046595 20178 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:52:01.050064 20245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:01.057137 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 20178
I20260806 04:52:01.057364 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:52:01.058002 13906 external_mini_cluster.cc:1428] Setting key 5e1531d362fb60c1e9ad2759d6dbd083
I20260806 04:52:01.059871 20245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b: Bootstrap starting.
I20260806 04:52:01.062995 20245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:01.064018 20245 log.cc:824] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b: Log is configured to *not* fsync() on all Append() calls
I20260806 04:52:01.066737 20245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b: No bootstrap required, opened a new log
Aug 06 04:52:01 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM
I20260806 04:52:01.073966 20245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } }
I20260806 04:52:01.074324 20245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:01.074440 20245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8545f2e2779a47e3ac5cb4398a291b1b, State: Initialized, Role: FOLLOWER
I20260806 04:52:01.074954 20245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } }
I20260806 04:52:01.075194 20245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:52:01.075352 20245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:52:01.075536 20245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:01.078510 20245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } }
I20260806 04:52:01.078980 20245 leader_election.cc:302] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8545f2e2779a47e3ac5cb4398a291b1b; no voters:
I20260806 04:52:01.079530 20245 leader_election.cc:288] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:52:01.079560 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:01.060001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.13.148.190:38833)
0806 04:52:01.060857 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:01.060869 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:01.060909 (+ 40us) server_negotiation.cc:407] Connection header received
0806 04:52:01.062483 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:01.062504 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:01.062860 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:01.063254 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:01.064742 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:01.066023 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:01.067284 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:01.067918 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:01.068152 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:01.072023 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:01.072034 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:01.072047 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:01.072091 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:01.075170 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:01.076098 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:01.076105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:01.076110 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:01.076273 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:01.076718 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:01.076722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:01.076724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:01.077311 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:01.077545 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:01.077991 (+ 446us) server_negotiation.cc:299] Negotiation successful
0806 04:52:01.078347 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":378,"threads_started":1}
I20260806 04:52:01.079669 20250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:52:01.081591 20245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:52:01.081681 20250 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [term 1 LEADER]: Becoming Leader. State: Replica: 8545f2e2779a47e3ac5cb4398a291b1b, State: Running, Role: LEADER
I20260806 04:52:01.082239 20250 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } }
I20260806 04:52:01.085423 20251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } }
I20260806 04:52:01.085258 20252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8545f2e2779a47e3ac5cb4398a291b1b. Latest consensus state: current_term: 1 leader_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } }
I20260806 04:52:01.085748 20252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:01.085748 20251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:01.087057 20259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:52:01.094517 20259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:52:01.102957 20259 catalog_manager.cc:1347] Generated new cluster ID: b888a0e01c47497bb126f3d75f810621
I20260806 04:52:01.103062 20259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:52:01.121536 20259 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:52:01.122922 20259 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:52:01.129665 20259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b: Generated new TSK 0
I20260806 04:52:01.130419 20259 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.129.keytab.
Entry for principal oryx/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.129.keytab.
I20260806 04:52:01.182806 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:38833
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.129.keytab
--principal=oryx/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:36971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:01.493124 20271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:01.493415 20271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:01.493561 20271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:01.499923 20271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:01.500082 20271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:01.500258 20271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:52:01.509119 20271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.129.keytab
--principal=oryx/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:38833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:01.510430 20271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:01.511780 20271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:01.524080 20276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:01.524057 20277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:01.524052 20279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:01.525178 20271 server_base.cc:1034] running on GCE node
I20260806 04:52:01.525728 20271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:01.526612 20271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:01.527827 20271 hybrid_clock.cc:648] HybridClock initialized: now 1785991921527787 us; error 32 us; skew 500 ppm
I20260806 04:52:01.528132 20271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:01 dist-test-slave-1x4g krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991921, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:01.531484 20271 init.cc:377] Logged in from keytab as oryx/127.13.148.129@KRBTEST.COM (short username oryx)
I20260806 04:52:01.532836 20271 webserver.cc:460] Webserver started at http://127.13.148.129:36681/ using document root <none> and password file <none>
I20260806 04:52:01.533504 20271 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:01.533627 20271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:01.533947 20271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:01.536840 20271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "75e962bb879145bd8a56e7cc23142fdf"
format_stamp: "Formatted at 2026-08-06 04:52:01 on dist-test-slave-1x4g"
server_key: "be0deb73a5077aa546238d3837a4c618"
server_key_iv: "4094a7bad04a7699c5328e7ff4db2e82"
server_key_version: "encryptionkey@0"
I20260806 04:52:01.537619 20271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "75e962bb879145bd8a56e7cc23142fdf"
format_stamp: "Formatted at 2026-08-06 04:52:01 on dist-test-slave-1x4g"
server_key: "be0deb73a5077aa546238d3837a4c618"
server_key_iv: "4094a7bad04a7699c5328e7ff4db2e82"
server_key_version: "encryptionkey@0"
I20260806 04:52:01.542400 20271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 04:52:01.545570 20286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:01.546764 20271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:52:01.546919 20271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "75e962bb879145bd8a56e7cc23142fdf"
format_stamp: "Formatted at 2026-08-06 04:52:01 on dist-test-slave-1x4g"
server_key: "be0deb73a5077aa546238d3837a4c618"
server_key_iv: "4094a7bad04a7699c5328e7ff4db2e82"
server_key_version: "encryptionkey@0"
I20260806 04:52:01.547088 20271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:01.564546 20271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:01.568261 20271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:01.568549 20271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:01.569478 20271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:01.570777 20271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:01.570883 20271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:01.571023 20271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:01.571092 20271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:01.602615 20271 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:33753
I20260806 04:52:01.602660 20402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:33753 every 8 connection(s)
I20260806 04:52:01.603890 20271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:52:01.612658 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 20271
I20260806 04:52:01.612915 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:52:01.613647 13906 external_mini_cluster.cc:1428] Setting key 9427c1598f2d508f6c09a7121d8eec32
Aug 06 04:52:01 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991921, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM
I20260806 04:52:01.624338 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:01.606410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:55109 (local address 127.13.148.190:38833)
0806 04:52:01.606813 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:01.606817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:01.607824 (+ 1007us) server_negotiation.cc:407] Connection header received
0806 04:52:01.609549 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:01.609554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:01.609619 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:01.609733 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:01.611137 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:01.611815 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:01.613944 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:01.614322 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:01.614443 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:01.617897 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:01.617906 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:01.617914 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:01.617966 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:01.621352 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:01.622012 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:01.622020 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:01.622026 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:01.622168 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:01.622538 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:01.622545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:01.622551 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:01.622915 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:01.623091 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:01.623727 (+ 636us) server_negotiation.cc:299] Negotiation successful
0806 04:52:01.623957 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":190,"threads_started":1}
I20260806 04:52:01.625523 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:01.607001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.129:55109)
0806 04:52:01.607661 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:01.607699 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:01.609284 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:01.609940 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:01.609948 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:01.610444 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:01.610983 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:01.611000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:01.611939 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:01.611942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:01.613791 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:01.613807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:01.614535 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:01.614540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:01.614777 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:01.615483 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:01.615517 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:01.617678 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:01.621483 (+ 3805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:01.621489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:01.621500 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:01.621885 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:01.622265 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:01.622267 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:01.622269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:01.622432 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:01.623068 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:01.623073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:01.623359 (+ 286us) client_negotiation.cc:769] Sending connection context
0806 04:52:01.623625 (+ 266us) client_negotiation.cc:240] Negotiation successful
0806 04:52:01.624157 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":250,"threads_started":1}
I20260806 04:52:01.626946 20403 heartbeater.cc:347] Connected to a master server at 127.13.148.190:38833
I20260806 04:52:01.627259 20403 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:01.628150 20403 heartbeater.cc:511] Master 127.13.148.190:38833 requested a full tablet report, sending...
I20260806 04:52:01.630710 20210 ts_manager.cc:194] Registered new tserver with Master: 75e962bb879145bd8a56e7cc23142fdf (127.13.148.129:33753)
I20260806 04:52:01.632849 20210 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:55109
WARNING: no policy specified for oryx/127.13.148.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.130.keytab.
Entry for principal oryx/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.130.keytab.
I20260806 04:52:01.665012 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:38833
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.130.keytab
--principal=oryx/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:36971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:01.975543 20409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:01.975847 20409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:01.976024 20409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:01.983167 20409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:01.983278 20409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:01.983422 20409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:52:01.992837 20409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.130.keytab
--principal=oryx/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:38833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:01.994138 20409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:01.995404 20409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:02.008040 20414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:02.008001 20415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:02.008071 20417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:02.009068 20409 server_base.cc:1034] running on GCE node
I20260806 04:52:02.009630 20409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:02.010552 20409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:02.011788 20409 hybrid_clock.cc:648] HybridClock initialized: now 1785991922011752 us; error 54 us; skew 500 ppm
I20260806 04:52:02.012111 20409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:02 dist-test-slave-1x4g krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991922, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:02.015161 20409 init.cc:377] Logged in from keytab as oryx/127.13.148.130@KRBTEST.COM (short username oryx)
I20260806 04:52:02.016541 20409 webserver.cc:460] Webserver started at http://127.13.148.130:32853/ using document root <none> and password file <none>
I20260806 04:52:02.017293 20409 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:02.017434 20409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:02.017740 20409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:02.020584 20409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "06e4f08b79e043fcae6f7a6ad7b8f281"
format_stamp: "Formatted at 2026-08-06 04:52:02 on dist-test-slave-1x4g"
server_key: "61047ff539180177448e4c55bbfe1214"
server_key_iv: "23f1d2b7b83958600ce7caee00e10006"
server_key_version: "encryptionkey@0"
I20260806 04:52:02.021437 20409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "06e4f08b79e043fcae6f7a6ad7b8f281"
format_stamp: "Formatted at 2026-08-06 04:52:02 on dist-test-slave-1x4g"
server_key: "61047ff539180177448e4c55bbfe1214"
server_key_iv: "23f1d2b7b83958600ce7caee00e10006"
server_key_version: "encryptionkey@0"
I20260806 04:52:02.026402 20409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 04:52:02.030242 20424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:02.031564 20409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:52:02.031729 20409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "06e4f08b79e043fcae6f7a6ad7b8f281"
format_stamp: "Formatted at 2026-08-06 04:52:02 on dist-test-slave-1x4g"
server_key: "61047ff539180177448e4c55bbfe1214"
server_key_iv: "23f1d2b7b83958600ce7caee00e10006"
server_key_version: "encryptionkey@0"
I20260806 04:52:02.031919 20409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:02.054111 20409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:02.058156 20409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:02.058413 20409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:02.059341 20409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:02.060626 20409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:02.060730 20409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:02.060889 20409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:02.060967 20409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:02.092744 20409 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:46089
I20260806 04:52:02.092801 20540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:46089 every 8 connection(s)
I20260806 04:52:02.094055 20409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:52:02.094677 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 20409
I20260806 04:52:02.094887 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:52:02.095391 13906 external_mini_cluster.cc:1428] Setting key 4b2e55df13322b5d6ea4667f91d4383e
Aug 06 04:52:02 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991922, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM
I20260806 04:52:02.114858 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:02.097025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:60879 (local address 127.13.148.190:38833)
0806 04:52:02.097204 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:02.097212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:02.098617 (+ 1405us) server_negotiation.cc:407] Connection header received
0806 04:52:02.100569 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:02.100574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:02.100666 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:02.100768 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:02.102276 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:02.103009 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:02.104483 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:02.104927 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:02.105061 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:02.108938 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:02.108943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:02.108946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:02.108978 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:02.111633 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:02.112561 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:02.112565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:02.112568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:02.112688 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:02.113181 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:02.113184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:02.113186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:02.113429 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:02.113576 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:02.114454 (+ 878us) server_negotiation.cc:299] Negotiation successful
0806 04:52:02.114616 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:52:02.116492 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:02.097682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.130:60879)
0806 04:52:02.098460 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:02.098497 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:02.100311 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:02.101037 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:02.101045 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:02.101539 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:02.102075 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:02.102092 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:02.103153 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:02.103163 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:02.104340 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:02.104352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:02.105153 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:02.105160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:02.105373 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:02.106088 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:02.106114 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:02.108680 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:02.111773 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:02.111785 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:02.111803 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:02.112437 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:02.112817 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:02.112824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:02.112831 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:02.113073 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:02.113570 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:02.113580 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:02.114050 (+ 470us) client_negotiation.cc:769] Sending connection context
0806 04:52:02.114362 (+ 312us) client_negotiation.cc:240] Negotiation successful
0806 04:52:02.114978 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":317,"threads_started":1}
I20260806 04:52:02.118083 20541 heartbeater.cc:347] Connected to a master server at 127.13.148.190:38833
WARNING: no policy specified for oryx/127.13.148.131@KRBTEST.COM; defaulting to no policy
I20260806 04:52:02.118392 20541 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:02.119103 20541 heartbeater.cc:511] Master 127.13.148.190:38833 requested a full tablet report, sending...
I20260806 04:52:02.120499 20210 ts_manager.cc:194] Registered new tserver with Master: 06e4f08b79e043fcae6f7a6ad7b8f281 (127.13.148.130:46089)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.131@KRBTEST.COM" created.
I20260806 04:52:02.121488 20210 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:60879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.131.keytab.
Entry for principal oryx/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.131.keytab.
I20260806 04:52:02.146128 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:38833
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.131.keytab
--principal=oryx/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:36971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:02.456154 20546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:02.456454 20546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:02.456643 20546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:02.463733 20546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:02.463901 20546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:02.464094 20546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:52:02.473712 20546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.131.keytab
--principal=oryx/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:38833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:02.475011 20546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:02.476356 20546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:02.489387 20551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:02.489529 20552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:02.489470 20554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:02.490216 20546 server_base.cc:1034] running on GCE node
I20260806 04:52:02.490808 20546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:02.491679 20546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:02.492885 20546 hybrid_clock.cc:648] HybridClock initialized: now 1785991922492834 us; error 24 us; skew 500 ppm
I20260806 04:52:02.493191 20546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:02 dist-test-slave-1x4g krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991922, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:02.499516 20546 init.cc:377] Logged in from keytab as oryx/127.13.148.131@KRBTEST.COM (short username oryx)
I20260806 04:52:02.501171 20546 webserver.cc:460] Webserver started at http://127.13.148.131:40339/ using document root <none> and password file <none>
I20260806 04:52:02.501855 20546 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:02.501992 20546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:02.502290 20546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:02.505251 20546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "b39b7889a4cd4aa781f52d643dec6432"
format_stamp: "Formatted at 2026-08-06 04:52:02 on dist-test-slave-1x4g"
server_key: "857f15d9d36934225832b3d5dc110503"
server_key_iv: "d02c46a5795cf344c1339ead7f7adb32"
server_key_version: "encryptionkey@0"
I20260806 04:52:02.506021 20546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "b39b7889a4cd4aa781f52d643dec6432"
format_stamp: "Formatted at 2026-08-06 04:52:02 on dist-test-slave-1x4g"
server_key: "857f15d9d36934225832b3d5dc110503"
server_key_iv: "d02c46a5795cf344c1339ead7f7adb32"
server_key_version: "encryptionkey@0"
I20260806 04:52:02.510927 20546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 04:52:02.514456 20561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:02.515719 20546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:52:02.515903 20546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "b39b7889a4cd4aa781f52d643dec6432"
format_stamp: "Formatted at 2026-08-06 04:52:02 on dist-test-slave-1x4g"
server_key: "857f15d9d36934225832b3d5dc110503"
server_key_iv: "d02c46a5795cf344c1339ead7f7adb32"
server_key_version: "encryptionkey@0"
I20260806 04:52:02.516101 20546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:02.532020 20546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:02.535804 20546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:02.536109 20546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:02.537101 20546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:02.538478 20546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:02.538582 20546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:02.538733 20546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:02.538810 20546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:02.572875 20546 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:46529
I20260806 04:52:02.572885 20677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:46529 every 8 connection(s)
I20260806 04:52:02.574273 20546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:52:02.575469 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 20546
I20260806 04:52:02.575677 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:52:02.576211 13906 external_mini_cluster.cc:1428] Setting key af553ff3f9431e08721899fff63b2f29
Aug 06 04:52:02 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991922, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM
I20260806 04:52:02.593753 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:02.577114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:40977 (local address 127.13.148.190:38833)
0806 04:52:02.577297 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:02.577301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:02.578730 (+ 1429us) server_negotiation.cc:407] Connection header received
0806 04:52:02.580601 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:02.580605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:02.580668 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:02.580746 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:02.582309 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:02.583078 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:02.584513 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:02.584975 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:02.585061 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:02.588186 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:02.588191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:02.588194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:02.588225 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:02.591311 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:02.591985 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:02.591989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:02.591991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:02.592104 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:02.592442 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:02.592445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:02.592447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:02.592706 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:02.592852 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:02.593367 (+ 515us) server_negotiation.cc:299] Negotiation successful
0806 04:52:02.593516 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 04:52:02.594647 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:02.577770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.131:40977)
0806 04:52:02.578565 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:02.578612 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:02.580356 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:02.580995 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:02.581003 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:02.581614 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:02.582186 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:02.582203 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:02.583223 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:02.583226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:02.584393 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:02.584404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:02.585136 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:02.585140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:02.585318 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:02.585894 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:02.585914 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:02.587995 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:02.591447 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:02.591455 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:02.591470 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:02.591878 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:02.592188 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:02.592191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:02.592193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:02.592360 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:02.592827 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:02.592831 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:02.593099 (+ 268us) client_negotiation.cc:769] Sending connection context
0806 04:52:02.593304 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 04:52:02.593641 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":294,"threads_started":1}
I20260806 04:52:02.595866 20678 heartbeater.cc:347] Connected to a master server at 127.13.148.190:38833
I20260806 04:52:02.596136 20678 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:02.596868 20678 heartbeater.cc:511] Master 127.13.148.190:38833 requested a full tablet report, sending...
I20260806 04:52:02.598240 20210 ts_manager.cc:194] Registered new tserver with Master: b39b7889a4cd4aa781f52d643dec6432 (127.13.148.131:46529)
I20260806 04:52:02.598963 20210 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:40977
I20260806 04:52:02.603581 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.148.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.189@KRBTEST.COM" created.
I20260806 04:52:02.636241 20403 heartbeater.cc:503] Master 127.13.148.190:38833 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.189.keytab.
Entry for principal oryx/127.13.148.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.189.keytab.
I20260806 04:52:02.649844 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.189:40159
--webserver_interface=127.13.148.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.189.keytab
--principal=oryx/127.13.148.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:36971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:38833,127.13.148.189:40159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:02.966902 20683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:02.967211 20683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:02.967358 20683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:02.974186 20683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:52:02.974298 20683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:02.974395 20683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:02.974478 20683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:52:02.974552 20683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:52:02.983476 20683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:38833,127.13.148.189:40159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.189.keytab
--principal=oryx/127.13.148.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.189:40159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.148.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
W20260806 04:52:02.983786 20683 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.13.148.190:38833,127.13.148.189:40159'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 04:52:02.984652 20683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:02.985855 20683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:02.996171 20688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:02.996199 20689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:02.996243 20691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:02.996963 20683 server_base.cc:1034] running on GCE node
I20260806 04:52:02.997689 20683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:02.998858 20683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:03.000089 20683 hybrid_clock.cc:648] HybridClock initialized: now 1785991923000010 us; error 46 us; skew 500 ppm
I20260806 04:52:03.000397 20683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:02 dist-test-slave-1x4g krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991922, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:03.003369 20683 init.cc:377] Logged in from keytab as oryx/127.13.148.189@KRBTEST.COM (short username oryx)
I20260806 04:52:03.004591 20683 webserver.cc:460] Webserver started at http://127.13.148.189:44617/ using document root <none> and password file <none>
I20260806 04:52:03.005223 20683 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:03.005303 20683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:03.005573 20683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:03.008131 20683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/data/instance:
uuid: "324249b33cc748309596dc8e2ec5f26e"
format_stamp: "Formatted at 2026-08-06 04:52:03 on dist-test-slave-1x4g"
server_key: "7f372fdf8d5379d18b4e018e987c7ecd"
server_key_iv: "5923090d350b5fae45d595aa3c1f1fcc"
server_key_version: "encryptionkey@0"
I20260806 04:52:03.008834 20683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/wal/instance:
uuid: "324249b33cc748309596dc8e2ec5f26e"
format_stamp: "Formatted at 2026-08-06 04:52:03 on dist-test-slave-1x4g"
server_key: "7f372fdf8d5379d18b4e018e987c7ecd"
server_key_iv: "5923090d350b5fae45d595aa3c1f1fcc"
server_key_version: "encryptionkey@0"
I20260806 04:52:03.013626 20683 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 04:52:03.016677 20698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:03.017972 20683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:52:03.018114 20683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/wal
uuid: "324249b33cc748309596dc8e2ec5f26e"
format_stamp: "Formatted at 2026-08-06 04:52:03 on dist-test-slave-1x4g"
server_key: "7f372fdf8d5379d18b4e018e987c7ecd"
server_key_iv: "5923090d350b5fae45d595aa3c1f1fcc"
server_key_version: "encryptionkey@0"
I20260806 04:52:03.018304 20683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:03.075556 20683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:03.079849 20683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:03.080114 20683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:03.098889 20683 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.189:40159
I20260806 04:52:03.098923 20752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.189:40159 every 8 connection(s)
I20260806 04:52:03.100226 20683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/data/info.pb
I20260806 04:52:03.103847 20753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:03.108708 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 20683
I20260806 04:52:03.110425 20753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:52:03 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991922, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.189@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM
Aug 06 04:52:03 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.189@KRBTEST.COM
I20260806 04:52:03.124666 20541 heartbeater.cc:503] Master 127.13.148.190:38833 was elected leader, sending a full tablet report...
I20260806 04:52:03.128532 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.112997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.13.148.190:38833)
0806 04:52:03.113437 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.113442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.113969 (+ 527us) server_negotiation.cc:407] Connection header received
0806 04:52:03.114331 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.114338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.114408 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.114535 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:03.115709 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.116458 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.118009 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.118390 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.118518 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.122347 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:03.122355 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:03.122361 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:03.122399 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:03.125450 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.126401 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.126405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.126407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.126529 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.126934 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.126938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.126940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.127215 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:03.127356 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.127982 (+ 626us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.128182 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":260,"threads_started":1}
I20260806 04:52:03.129614 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.113077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:50440)
0806 04:52:03.113840 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:03.113875 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:03.114133 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:03.114628 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:03.114634 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:03.115202 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:03.115549 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.115565 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.116618 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.116621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.117873 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.117883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.118605 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.118612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.118787 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.119516 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:03.119536 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:03.122171 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:03.125596 (+ 3425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.125606 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.125615 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.126153 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.126629 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.126633 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.126637 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.126823 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.127332 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:03.127339 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:03.127650 (+ 311us) client_negotiation.cc:769] Sending connection context
0806 04:52:03.127896 (+ 246us) client_negotiation.cc:240] Negotiation successful
0806 04:52:03.128325 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":31,"thread_start_us":476,"threads_started":1}
I20260806 04:52:03.129633 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.110866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47766 (local address 127.13.148.189:40159)
0806 04:52:03.111658 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.111669 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.111708 (+ 39us) server_negotiation.cc:407] Connection header received
0806 04:52:03.113944 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.113966 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.114238 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.114425 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:03.115837 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.117027 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.118237 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.118704 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.118889 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.123701 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:03.123707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:03.123716 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:03.123744 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:03.126869 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.127489 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.127500 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.127502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.127620 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.127963 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.127969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.127973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.128245 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:03.128533 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.129007 (+ 474us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.129218 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":614,"spinlock_wait_cycles":12672,"thread_start_us":385,"threads_started":1}
I20260806 04:52:03.131342 20753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:52:03 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991922, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.189@KRBTEST.COM for oryx/127.13.148.189@KRBTEST.COM
I20260806 04:52:03.142066 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.131899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:40159 (local address 127.0.0.1:47772)
0806 04:52:03.132099 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:03.132117 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:03.132201 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:03.132471 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:03.132474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:03.132726 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:03.132898 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.132906 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.133683 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.133686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.134380 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.134389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.134944 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.134947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.135071 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.135528 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:03.135542 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:03.137381 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:03.140215 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.140218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.140221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.140572 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.140923 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.140926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.140928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.141053 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.141497 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:03.141499 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:03.141587 (+ 88us) client_negotiation.cc:769] Sending connection context
0806 04:52:03.141685 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 04:52:03.141857 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 04:52:03.142259 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.132059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.13.148.189:40159)
0806 04:52:03.132205 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.132209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.132224 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:52:03.132289 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.132295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.132351 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.132437 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:03.132998 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.133584 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.134516 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.134843 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.134923 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.137556 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:03.137562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:03.137566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:03.137603 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:03.140070 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.140662 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.140666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.140668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.140838 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.141138 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.141141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.141143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.141405 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:03.141506 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.141758 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.141959 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":61}
I20260806 04:52:03.146071 20753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: Bootstrap starting.
I20260806 04:52:03.148661 20753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:03.149633 20753 log.cc:824] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: Log is configured to *not* fsync() on all Append() calls
I20260806 04:52:03.152048 20753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: No bootstrap required, opened a new log
I20260806 04:52:03.158365 20753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } }
I20260806 04:52:03.158705 20753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:03.158805 20753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324249b33cc748309596dc8e2ec5f26e, State: Initialized, Role: FOLLOWER
I20260806 04:52:03.159375 20753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } }
I20260806 04:52:03.160158 20760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } } }
I20260806 04:52:03.160439 20760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:52:03.161147 20753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: configured and running, proceeding with master startup.
W20260806 04:52:03.168587 20772 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:52:03.168708 20772 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:52:03.170487 20683 master_runner.cc:428] Detected that this master 324249b33cc748309596dc8e2ec5f26e is joining an existing cluster
I20260806 04:52:03.170589 20683 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.148.189:40159
I20260806 04:52:03.189318 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.174129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47778 (local address 127.13.148.189:40159)
0806 04:52:03.174325 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.174334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.175318 (+ 984us) server_negotiation.cc:407] Connection header received
0806 04:52:03.175475 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.175480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.175545 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.175606 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:03.177132 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.178085 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.179385 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.180113 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.180236 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.182854 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:03.182860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:03.182864 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:03.182899 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:03.186320 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.187097 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.187103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.187106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.187261 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.187664 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.187668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.187671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.188021 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:03.188131 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.188679 (+ 548us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.188935 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 04:52:03.189431 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.174134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50442 (local address 127.13.148.190:38833)
0806 04:52:03.174346 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.174352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.175052 (+ 700us) server_negotiation.cc:407] Connection header received
0806 04:52:03.175275 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.175279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.175353 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.175436 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:03.176530 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.177554 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.179570 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.180229 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.180343 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.182602 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:03.182606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:03.182609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:03.182657 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:03.186099 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.186927 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.186934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.186939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.187110 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.187540 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.187546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.187551 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.187883 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:03.188027 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.188692 (+ 665us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.188990 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 04:52:03.190773 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.174241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:50442)
0806 04:52:03.174935 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:03.174960 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:03.175154 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:03.175467 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:03.175471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:03.176058 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:03.176399 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.176408 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.177700 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.177710 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.179429 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.179448 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.180560 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.180564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.180704 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.181400 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:03.181415 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:03.182389 (+ 974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:03.186237 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.186244 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.186254 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.186799 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.187256 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.187261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.187266 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.187425 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.188121 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:03.188127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:03.188327 (+ 200us) client_negotiation.cc:769] Sending connection context
0806 04:52:03.189692 (+ 1365us) client_negotiation.cc:240] Negotiation successful
0806 04:52:03.190148 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":42,"thread_start_us":377,"threads_started":1}
I20260806 04:52:03.191720 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.174545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:40159 (local address 127.0.0.1:47778)
0806 04:52:03.175242 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:03.175259 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:03.175367 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:03.175789 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:03.175793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:03.176588 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:03.176976 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.176989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.178235 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.178239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.179235 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.179247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.180302 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.180308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.180483 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.181175 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:03.181211 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:03.182421 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:03.186451 (+ 4030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.186455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.186458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.186987 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.187388 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.187392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.187395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.187560 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.188206 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:03.188210 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:03.188355 (+ 145us) client_negotiation.cc:769] Sending connection context
0806 04:52:03.189915 (+ 1560us) client_negotiation.cc:240] Negotiation successful
0806 04:52:03.190344 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":203,"spinlock_wait_cycles":19712,"thread_start_us":430,"threads_started":1}
Aug 06 04:52:03 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991920, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.190@KRBTEST.COM for oryx/127.13.148.189@KRBTEST.COM
WARNING: no policy specified for oryx/127.13.148.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.188@KRBTEST.COM" created.
I20260806 04:52:03.212415 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.197290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.13.148.189:40159)
0806 04:52:03.197449 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.197455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.198143 (+ 688us) server_negotiation.cc:407] Connection header received
0806 04:52:03.198372 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.198376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.198463 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.198555 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:03.199454 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.200331 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.201756 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.202359 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.202462 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.206197 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:03.206204 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:03.206209 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:03.206245 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:03.209489 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.210283 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.210290 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.210297 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.210439 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.210880 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.210887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.210892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.211284 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:03.211449 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.211848 (+ 399us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.212050 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:52:03.212487 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.197395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:40159 (local address 127.0.0.1:47784)
0806 04:52:03.198022 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:03.198053 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:03.198196 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:03.198597 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:03.198602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:03.199146 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:03.199326 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.199337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.200442 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.200445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.201603 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.201617 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.202521 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.202527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.202755 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.203383 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:03.203406 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:03.206010 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:03.209604 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.209616 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.209622 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.210157 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.210554 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.210560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.210563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.210761 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.211425 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:03.211433 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:03.211599 (+ 166us) client_negotiation.cc:769] Sending connection context
0806 04:52:03.211767 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 04:52:03.212068 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":269,"threads_started":1}
I20260806 04:52:03.214370 20210 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.148.189:40159: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "324249b33cc748309596dc8e2ec5f26e"
member_type: NON_VOTER
last_known_addr {
host: "127.13.148.189"
port: 40159
}
attrs {
promote: true
}
}
dest_uuid: "8545f2e2779a47e3ac5cb4398a291b1b"
cas_config_opid_index: -1
I20260806 04:52:03.215348 20210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: NON_VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: true } }
I20260806 04:52:03.220213 20785 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 324249b33cc748309596dc8e2ec5f26e (127.13.148.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: NON_VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: true } } }
I20260806 04:52:03.221333 20728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:03.224611 20785 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.148.189:40159
I20260806 04:52:03.225997 20728 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 FOLLOWER]: Refusing update from remote peer 8545f2e2779a47e3ac5cb4398a291b1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 04:52:03.226402 20683 master.cc:537] Master@127.13.148.189:40159 shutting down...
I20260806 04:52:03.233098 20783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: NON_VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: true } } }
I20260806 04:52:03.233402 20783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:03.234167 20786 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: NON_VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 04:52:03.237248 20197 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 04:52:03.239169 20197 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b -> Peer 324249b33cc748309596dc8e2ec5f26e (127.13.148.189:40159): Couldn't send request to peer 324249b33cc748309596dc8e2ec5f26e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:52:03.239240 20683 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:52:03.239596 20683 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:03.239794 20683 tablet_replica.cc:330] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: stopping tablet replica
I20260806 04:52:03.245172 20683 master.cc:552] Master@127.13.148.189:40159 shutdown complete.
I20260806 04:52:03.245253 20683 master_runner.cc:305] Clearing existing system tablet
I20260806 04:52:03.245579 20683 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:03.245668 20683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:03.247582 20683 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 04:52:03.248889 20790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:03.249161 20683 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:03.249404 20683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 04:52:03.249483 20683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/wal
uuid: "324249b33cc748309596dc8e2ec5f26e"
format_stamp: "Formatted at 2026-08-06 04:52:03 on dist-test-slave-1x4g"
server_key: "7f372fdf8d5379d18b4e018e987c7ecd"
server_key_iv: "5923090d350b5fae45d595aa3c1f1fcc"
server_key_version: "encryptionkey@0"
I20260806 04:52:03.252981 20683 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:03.258553 20683 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 04:52:03.258724 20683 log.cc:1196] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.188.keytab.
Entry for principal oryx/127.13.148.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.188.keytab.
I20260806 04:52:03.259107 20683 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: Deleting consensus metadata
I20260806 04:52:03.259333 20683 master_runner.cc:315] Copying system tablet from 127.13.148.190:38833
I20260806 04:52:03.260502 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.188:39045
--webserver_interface=127.13.148.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.188.keytab
--principal=oryx/127.13.148.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:36971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:38833,127.13.148.189:40159,127.13.148.188:39045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:52:03.265187 20683 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: tablet copy: Beginning tablet copy session from remote peer at address 127.13.148.190:38833
I20260806 04:52:03.277148 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.265971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.13.148.190:38833)
0806 04:52:03.266144 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.266150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.266454 (+ 304us) server_negotiation.cc:407] Connection header received
0806 04:52:03.266644 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.266649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.266716 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.266805 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:03.267726 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.268766 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.269866 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.270310 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.270450 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.272268 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:03.272273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:03.272277 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:03.272315 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:03.274724 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.275289 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.275295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.275298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.275419 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.275746 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.275751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.275754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.276006 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:03.276149 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.276517 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.276735 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:52:03.277202 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.265936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:50444)
0806 04:52:03.266351 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:03.266370 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:03.266515 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:03.266828 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:03.266832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:03.267266 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:03.267540 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.267553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.268909 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.268914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.269744 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.269754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.270477 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.270483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.270691 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.271353 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:03.271368 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:03.272115 (+ 747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:03.274845 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.274851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.274855 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.275184 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.275522 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.275527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.275530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.275651 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.276117 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:03.276121 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:03.276272 (+ 151us) client_negotiation.cc:769] Sending connection context
0806 04:52:03.276432 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 04:52:03.276734 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":231,"threads_started":1}
I20260806 04:52:03.278272 20230 tablet_copy_service.cc:140] P 8545f2e2779a47e3ac5cb4398a291b1b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 324249b33cc748309596dc8e2ec5f26e ({username='oryx', principal='oryx/127.13.148.189@KRBTEST.COM'} at 127.0.0.1:50444)
I20260806 04:52:03.278528 20230 tablet_copy_service.cc:161] P 8545f2e2779a47e3ac5cb4398a291b1b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 324249b33cc748309596dc8e2ec5f26e at {username='oryx', principal='oryx/127.13.148.189@KRBTEST.COM'} at 127.0.0.1:50444: session id = 324249b33cc748309596dc8e2ec5f26e-00000000000000000000000000000000
I20260806 04:52:03.281442 20230 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:52:03.283658 20683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 04:52:03.288331 20683 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: tablet copy: Starting download of 0 data blocks...
I20260806 04:52:03.288630 20683 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: tablet copy: Starting download of 1 WAL segments...
I20260806 04:52:03.290980 20683 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:52:03.294004 20230 tablet_copy_service.cc:342] P 8545f2e2779a47e3ac5cb4398a291b1b: Request end of tablet copy session 324249b33cc748309596dc8e2ec5f26e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.148.189@KRBTEST.COM'} at 127.0.0.1:50444
I20260806 04:52:03.294166 20230 tablet_copy_service.cc:433] P 8545f2e2779a47e3ac5cb4398a291b1b: ending tablet copy session 324249b33cc748309596dc8e2ec5f26e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 324249b33cc748309596dc8e2ec5f26e
I20260806 04:52:03.304008 20683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:03.306409 20798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:03.317426 20799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:03.317631 20801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:03.318495 20683 server_base.cc:1034] running on GCE node
I20260806 04:52:03.318889 20683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:03.319499 20683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:03.320837 20683 hybrid_clock.cc:648] HybridClock initialized: now 1785991923320743 us; error 34 us; skew 500 ppm
I20260806 04:52:03.321071 20683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:03 dist-test-slave-1x4g krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991923, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:03.323017 20683 init.cc:377] Logged in from keytab as oryx/127.13.148.189@KRBTEST.COM (short username oryx)
I20260806 04:52:03.324124 20683 webserver.cc:460] Webserver started at http://127.13.148.189:40641/ using document root <none> and password file <none>
I20260806 04:52:03.324402 20683 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:03.324479 20683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:03.326611 20683 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260806 04:52:03.328217 20809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:03.328576 20683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260806 04:52:03.328729 20683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/wal
uuid: "324249b33cc748309596dc8e2ec5f26e"
format_stamp: "Formatted at 2026-08-06 04:52:03 on dist-test-slave-1x4g"
server_key: "7f372fdf8d5379d18b4e018e987c7ecd"
server_key_iv: "5923090d350b5fae45d595aa3c1f1fcc"
server_key_version: "encryptionkey@0"
I20260806 04:52:03.328959 20683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:03.335592 20683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:03.338800 20683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:03.370738 20683 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.189:40159
I20260806 04:52:03.370761 20863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.189:40159 every 8 connection(s)
I20260806 04:52:03.371871 20683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-1/data/info.pb
I20260806 04:52:03.378618 20864 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:52:03.383014 20864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: Bootstrap starting.
I20260806 04:52:03.395809 20864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:52:03.396198 20864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: Bootstrap complete.
I20260806 04:52:03.397068 20864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: NON_VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: true } }
I20260806 04:52:03.397284 20864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 324249b33cc748309596dc8e2ec5f26e, State: Initialized, Role: LEARNER
I20260806 04:52:03.397537 20864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: NON_VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: true } }
I20260806 04:52:03.398330 20867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: NON_VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: true } } }
I20260806 04:52:03.398595 20864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:52:03.398684 20867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: This master's current role is: LEARNER
I20260806 04:52:03.406160 20878 catalog_manager.cc:1259] Loaded cluster ID: b888a0e01c47497bb126f3d75f810621
I20260806 04:52:03.406247 20878 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: loading cluster ID for follower catalog manager: success
I20260806 04:52:03.408340 20878 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: acquiring CA information for follower catalog manager: success
I20260806 04:52:03.410006 20878 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:52:03.601680 20678 heartbeater.cc:503] Master 127.13.148.190:38833 was elected leader, sending a full tablet report...
W20260806 04:52:03.629618 20797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:03.629915 20797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:03.630060 20797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:03.637248 20797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:52:03.637355 20797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:03.637470 20797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:03.637557 20797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:52:03.637626 20797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:52:03.646791 20797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:38833,127.13.148.189:40159,127.13.148.188:39045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.188.keytab
--principal=oryx/127.13.148.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.188:39045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.148.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:03.648025 20797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:03.649279 20797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:52:03.659828 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.652336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:40159 (local address 127.0.0.1:47786)
0806 04:52:03.652534 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:03.652553 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:03.652673 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:03.653498 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:03.653503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:03.653582 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:03.653831 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.653847 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.655979 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.655984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.657605 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.657619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.659301 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.659305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.659435 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.659476 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 04:52:03.659532 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 04:52:03.659616 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 04:52:03.660061 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.652617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.13.148.189:40159)
0806 04:52:03.653185 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.653189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.653204 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:52:03.653260 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.653263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.653321 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.653407 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:03.654088 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.655792 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.657744 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.659146 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.659360 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.659555 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.659611 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.659729 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":339,"threads_started":1}
I20260806 04:52:03.661362 20839 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260806 04:52:03.661963 20867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8545f2e2779a47e3ac5cb4398a291b1b. Latest consensus state: current_term: 1 leader_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: NON_VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: true } } }
I20260806 04:52:03.662261 20867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: This master's current role is: LEARNER
W20260806 04:52:03.663348 20884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:03.663440 20887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:03.663470 20885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:03.664347 20797 server_base.cc:1034] running on GCE node
I20260806 04:52:03.665086 20797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:03.666342 20797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:03.667577 20797 hybrid_clock.cc:648] HybridClock initialized: now 1785991923667504 us; error 39 us; skew 500 ppm
I20260806 04:52:03.667945 20797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:03 dist-test-slave-1x4g krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991923, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:03.670827 20797 init.cc:377] Logged in from keytab as oryx/127.13.148.188@KRBTEST.COM (short username oryx)
I20260806 04:52:03.672269 20797 webserver.cc:460] Webserver started at http://127.13.148.188:43579/ using document root <none> and password file <none>
I20260806 04:52:03.672991 20797 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:03.673132 20797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:03.673390 20797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:03.676280 20797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/data/instance:
uuid: "14c97b2918954c2c90029c4234e84213"
format_stamp: "Formatted at 2026-08-06 04:52:03 on dist-test-slave-1x4g"
server_key: "e24d90fbb9123ec418566d91fcb14697"
server_key_iv: "76cc420d4fd2eefc12a0b9f4952db19b"
server_key_version: "encryptionkey@0"
I20260806 04:52:03.677181 20797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/wal/instance:
uuid: "14c97b2918954c2c90029c4234e84213"
format_stamp: "Formatted at 2026-08-06 04:52:03 on dist-test-slave-1x4g"
server_key: "e24d90fbb9123ec418566d91fcb14697"
server_key_iv: "76cc420d4fd2eefc12a0b9f4952db19b"
server_key_version: "encryptionkey@0"
I20260806 04:52:03.682124 20797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260806 04:52:03.685530 20894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:03.686754 20797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:52:03.686923 20797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/wal
uuid: "14c97b2918954c2c90029c4234e84213"
format_stamp: "Formatted at 2026-08-06 04:52:03 on dist-test-slave-1x4g"
server_key: "e24d90fbb9123ec418566d91fcb14697"
server_key_iv: "76cc420d4fd2eefc12a0b9f4952db19b"
server_key_version: "encryptionkey@0"
I20260806 04:52:03.687182 20797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:03.702390 20797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:03.705550 20797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:03.705785 20797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:03.725095 20797 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.188:39045
I20260806 04:52:03.725123 20946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.188:39045 every 8 connection(s)
I20260806 04:52:03.726366 20797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/data/info.pb
I20260806 04:52:03.730028 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:03.733685 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 20797
I20260806 04:52:03.737017 20947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:52:03 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.188@KRBTEST.COM
Aug 06 04:52:03 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991923, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.188@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM
I20260806 04:52:03.754148 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.740926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.13.148.190:38833)
0806 04:52:03.741106 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.741112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.741794 (+ 682us) server_negotiation.cc:407] Connection header received
0806 04:52:03.742001 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.742006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.742073 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.742160 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:03.743101 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.743742 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.745117 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.745491 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.745611 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.748820 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:03.748827 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:03.748831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:03.748868 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:03.751590 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.752249 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.752255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.752258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.752376 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.752728 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.752733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.752737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.753018 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:03.753155 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.753616 (+ 461us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.753820 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:52:03.754926 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.735774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54786 (local address 127.13.148.188:39045)
0806 04:52:03.736581 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.736606 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.736671 (+ 65us) server_negotiation.cc:407] Connection header received
0806 04:52:03.738316 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.738369 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.738837 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.739220 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:03.740618 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.742264 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.743464 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.744036 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.744264 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.748558 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:03.748565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:03.748574 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:03.748606 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:03.751283 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.751949 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.751962 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.751965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.752109 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.752459 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.752462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.752464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.752949 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:03.753230 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.753552 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.753915 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":371,"threads_started":1}
I20260806 04:52:03.754944 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.740912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:50450)
0806 04:52:03.741656 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:03.741692 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:03.741841 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:03.742206 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:03.742209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:03.742727 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:03.742958 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.742967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.743886 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.743890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.744981 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.744993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.745713 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.745720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.745849 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.746440 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:03.746460 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:03.748602 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:03.751711 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.751723 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.751732 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.752130 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.752473 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.752477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.752481 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.752629 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.753138 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:03.753145 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:03.753308 (+ 163us) client_negotiation.cc:769] Sending connection context
0806 04:52:03.753551 (+ 243us) client_negotiation.cc:240] Negotiation successful
0806 04:52:03.754065 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":118,"thread_start_us":388,"threads_started":1}
I20260806 04:52:03.757011 20947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:52:03 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991923, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.188@KRBTEST.COM for oryx/127.13.148.189@KRBTEST.COM
I20260806 04:52:03.767453 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.757770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:40159 (local address 127.0.0.1:47796)
0806 04:52:03.757922 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:03.757947 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:03.758026 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:03.758283 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:03.758286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:03.758567 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:03.758674 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.758681 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.759482 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.759485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.760317 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.760328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.760999 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.761003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.761145 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.761624 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:03.761638 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:03.763233 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:03.765788 (+ 2555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.765790 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.765792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.766124 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.766404 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.766407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.766409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.766523 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.766900 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:03.766903 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:03.767004 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 04:52:03.767104 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 04:52:03.767258 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 04:52:03.767505 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.757906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.13.148.189:40159)
0806 04:52:03.758059 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.758062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.758077 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:52:03.758121 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.758125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.758175 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.758260 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:03.758781 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.759392 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.760432 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.760883 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.760981 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.763395 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:03.763399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:03.763402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:03.763429 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:03.765678 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.766210 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.766213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.766215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.766324 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.766597 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.766600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.766601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.766819 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:03.766924 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.767174 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.767316 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:52:03.768268 20947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:52:03 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991923, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.188@KRBTEST.COM for oryx/127.13.148.188@KRBTEST.COM
I20260806 04:52:03.778512 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.768845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:39045 (local address 127.0.0.1:54788)
0806 04:52:03.768951 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:03.768963 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:03.769025 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:03.769259 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:03.769261 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:03.769523 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:03.769634 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.769641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.770421 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.770424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.771204 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.771214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.771788 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.771792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.771937 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.772333 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:03.772346 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:03.773947 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:03.776541 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.776543 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.776545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.776974 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.777273 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.777276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.777278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.777424 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.777836 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:03.777839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:03.777952 (+ 113us) client_negotiation.cc:769] Sending connection context
0806 04:52:03.778057 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 04:52:03.778223 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260806 04:52:03.778518 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.768978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.13.148.188:39045)
0806 04:52:03.769087 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.769090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.769102 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:52:03.769132 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.769134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.769172 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.769233 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:03.769730 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.770323 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.771331 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.771685 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.771764 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.774086 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:03.774090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:03.774092 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:03.774115 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:03.776427 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.777072 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.777076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.777078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.777188 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.777513 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.777516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.777518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.777749 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:03.777849 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.778180 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.778308 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 04:52:03.782852 20947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: Bootstrap starting.
I20260806 04:52:03.785760 20947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:03.786708 20947 log.cc:824] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: Log is configured to *not* fsync() on all Append() calls
I20260806 04:52:03.789052 20947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: No bootstrap required, opened a new log
I20260806 04:52:03.795603 20947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } }
I20260806 04:52:03.795886 20947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:03.795996 20947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c97b2918954c2c90029c4234e84213, State: Initialized, Role: FOLLOWER
I20260806 04:52:03.796455 20947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } }
I20260806 04:52:03.797256 20953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } } }
I20260806 04:52:03.797538 20953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:52:03.798115 20947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 04:52:03.807427 20964 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:52:03.807581 20964 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:52:03.809422 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 20271
I20260806 04:52:03.811564 20797 master_runner.cc:428] Detected that this master 14c97b2918954c2c90029c4234e84213 is joining an existing cluster
I20260806 04:52:03.811668 20797 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.148.188:39045
Aug 06 04:52:03 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.129@KRBTEST.COM
I20260806 04:52:03.829895 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.810628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33272 (local address 127.13.148.129:33753)
0806 04:52:03.811342 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.811349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.811373 (+ 24us) server_negotiation.cc:407] Connection header received
0806 04:52:03.811436 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.811443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.811610 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.811805 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:03.813159 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.814321 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.815582 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.816432 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.816537 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.822365 (+ 5828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:03.822377 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:03.822390 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:03.822426 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:03.826018 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.826809 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.826817 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.826820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.826976 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.827434 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.827439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.827441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.827821 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:03.828121 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.829278 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.829515 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":335,"threads_started":1}
I20260806 04:52:03.831498 20398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:03.832384 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.815679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:50462)
0806 04:52:03.816326 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:03.816349 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:03.816530 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:03.817079 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:03.817083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:03.817898 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:03.818224 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.818237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.819778 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.819783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.821773 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.821788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.823330 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.823342 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.823516 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.824061 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:03.824084 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:03.825195 (+ 1111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:03.829165 (+ 3970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.829170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.829173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.829646 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.830378 (+ 732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.830382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.830385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.830585 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.831249 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:03.831252 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:03.831437 (+ 185us) client_negotiation.cc:769] Sending connection context
0806 04:52:03.831673 (+ 236us) client_negotiation.cc:240] Negotiation successful
0806 04:52:03.831960 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":635,"threads_started":1}
I20260806 04:52:03.832389 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.815638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50462 (local address 127.13.148.190:38833)
0806 04:52:03.815786 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.815790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.816426 (+ 636us) server_negotiation.cc:407] Connection header received
0806 04:52:03.816700 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.816705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.816778 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.816877 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:03.818397 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.819588 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.821929 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.822575 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.822664 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.825450 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:03.825460 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:03.825464 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:03.825504 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:03.828716 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.829770 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.829774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.829776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.829912 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.830726 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.830731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.830734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.831072 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:03.831184 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.831843 (+ 659us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.832060 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:52:03.846783 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.815529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54804 (local address 127.13.148.188:39045)
0806 04:52:03.815696 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.815701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.827570 (+ 11869us) server_negotiation.cc:407] Connection header received
0806 04:52:03.827614 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.827620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.827685 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.827761 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:03.829496 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.830853 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.837026 (+ 6173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.837873 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.837966 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.840309 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:03.840314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:03.840318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:03.840361 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:03.844019 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.844704 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.844709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.844713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.844877 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.845285 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.845292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.845297 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.845639 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:03.845797 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.846258 (+ 461us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.846463 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":48}
Aug 06 04:52:03 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991920, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.190@KRBTEST.COM for oryx/127.13.148.188@KRBTEST.COM
I20260806 04:52:03.847579 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.816045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:39045 (local address 127.0.0.1:54804)
0806 04:52:03.827314 (+ 11269us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:03.827340 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:03.827472 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:03.827921 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:03.827927 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:03.828852 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:03.829323 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.829340 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.832563 (+ 3223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.832571 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.834502 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.834520 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.838079 (+ 3559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.838088 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.838295 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.839384 (+ 1089us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:03.839400 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:03.840105 (+ 705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:03.844140 (+ 4035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.844144 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.844147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.844607 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.845013 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.845017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.845021 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.845182 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.845859 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:03.845863 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:03.846013 (+ 150us) client_negotiation.cc:769] Sending connection context
0806 04:52:03.846928 (+ 915us) client_negotiation.cc:240] Negotiation successful
0806 04:52:03.847190 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11119,"thread_start_us":10955,"threads_started":1}
I20260806 04:52:03.856674 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.837565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:39045 (local address 127.0.0.1:54820)
0806 04:52:03.837873 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:03.837888 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:03.837982 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:03.839303 (+ 1321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:03.839309 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:03.839880 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:03.840074 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.840086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.842176 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.842182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.843487 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.843502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.844480 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.844485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.844629 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.845160 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:03.845179 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:03.848092 (+ 2913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:03.854099 (+ 6007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.854104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.854107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.854595 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.854973 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.854977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.854981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.855139 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.855771 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:03.855776 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:03.855943 (+ 167us) client_negotiation.cc:769] Sending connection context
0806 04:52:03.856063 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 04:52:03.856314 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260806 04:52:03.857800 20976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.838156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.13.148.188:39045)
0806 04:52:03.838901 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.838905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.838926 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:52:03.839024 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.839028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.839137 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.840257 (+ 1120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:03.840625 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.842027 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.843677 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.844303 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.849152 (+ 4849us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.849431 (+ 279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:03.849437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:03.849472 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:03.849539 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:03.853973 (+ 4434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.854718 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.854724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.854728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.854860 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.855273 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.855278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.855281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.855629 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:03.857091 (+ 1462us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.857207 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.857390 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":530,"threads_started":1}
I20260806 04:52:03.859067 20210 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.148.188:39045: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "14c97b2918954c2c90029c4234e84213"
member_type: NON_VOTER
last_known_addr {
host: "127.13.148.188"
port: 39045
}
attrs {
promote: true
}
}
dest_uuid: "8545f2e2779a47e3ac5cb4398a291b1b"
cas_config_opid_index: 5
I20260806 04:52:03.860116 20210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: NON_VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: true } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: NON_VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: true } }
I20260806 04:52:03.862360 20839 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 LEARNER]: Refusing update from remote peer 8545f2e2779a47e3ac5cb4398a291b1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 04:52:03.863561 20921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:03.867033 20921 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 1 FOLLOWER]: Refusing update from remote peer 8545f2e2779a47e3ac5cb4398a291b1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 04:52:03.867841 20786 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 14c97b2918954c2c90029c4234e84213 (127.13.148.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: NON_VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: true } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: NON_VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: true } } }
I20260806 04:52:03.869158 20978 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: NON_VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:52:03.869712 20978 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: NON_VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:52:03.880021 20786 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.148.188:39045
I20260806 04:52:03.882026 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.823027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:40159 (local address 127.0.0.1:47810)
0806 04:52:03.833310 (+ 10283us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:03.833327 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:03.833485 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:03.834083 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:03.834087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:03.834537 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:03.834718 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.834728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.836002 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.836007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.838849 (+ 2842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.838866 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.839967 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.839973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.840184 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.873361 (+ 33177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:03.873381 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:03.874347 (+ 966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:03.878619 (+ 4272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.878624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.878627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.879223 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.879692 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.879697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.879700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.879837 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.880784 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:03.880821 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:03.881026 (+ 205us) client_negotiation.cc:769] Sending connection context
0806 04:52:03.881184 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 04:52:03.881526 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10195,"mutex_wait_us":3,"thread_start_us":15191,"threads_started":1}
I20260806 04:52:03.882790 20797 master.cc:537] Master@127.13.148.188:39045 shutting down...
I20260806 04:52:03.883353 20786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: NON_VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: true } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: NON_VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: true } } }
I20260806 04:52:03.884543 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.822992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47810 (local address 127.13.148.189:40159)
0806 04:52:03.823160 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.823166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.833387 (+ 10221us) server_negotiation.cc:407] Connection header received
0806 04:52:03.833609 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.833614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.833689 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.833764 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:03.834868 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.835737 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.839027 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.839824 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.839950 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.874568 (+ 34618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:03.874574 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:03.874579 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:03.874615 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:03.878467 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.879360 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.879365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.879368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.879516 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.880188 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.880195 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.880199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.880552 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:03.880710 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.883850 (+ 3140us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.884186 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 04:52:03.884757 20786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:03.895804 20797 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:52:03.896462 20797 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:03.897042 20797 tablet_replica.cc:330] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: stopping tablet replica
W20260806 04:52:03.903115 20194 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b -> Peer 14c97b2918954c2c90029c4234e84213 (127.13.148.188:39045): Couldn't send request to peer 14c97b2918954c2c90029c4234e84213. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:52:03.904613 20839 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 14c97b2918954c2c90029c4234e84213 (127.13.148.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: NON_VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: true } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: NON_VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: true } } }
I20260806 04:52:03.906337 20797 master.cc:552] Master@127.13.148.188:39045 shutdown complete.
I20260806 04:52:03.906405 20797 master_runner.cc:305] Clearing existing system tablet
I20260806 04:52:03.906757 20797 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:03.906860 20797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:03.908805 20867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: NON_VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: true } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: NON_VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: true } } }
I20260806 04:52:03.909075 20867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: This master's current role is: LEARNER
I20260806 04:52:03.910367 20786 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b: attempting to promote NON_VOTER 324249b33cc748309596dc8e2ec5f26e to VOTER
I20260806 04:52:03.911231 20786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: NON_VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: true } }
I20260806 04:52:03.913059 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.912574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:39045 (local address 127.0.0.1:54836)
0806 04:52:03.912734 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:03.912881 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 04:52:03.913532 20839 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 LEARNER]: Refusing update from remote peer 8545f2e2779a47e3ac5cb4398a291b1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 04:52:03.914325 20786 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:52:03.914479 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.914171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:39045 (local address 127.0.0.1:54848)
0806 04:52:03.914270 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:03.914356 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260806 04:52:03.914770 20194 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b -> Peer 14c97b2918954c2c90029c4234e84213 (127.13.148.188:39045): Couldn't send request to peer 14c97b2918954c2c90029c4234e84213. Status: Network error: Client connection negotiation failed: client connection to 127.13.148.188:39045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 04:52:03.916364 20977 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 324249b33cc748309596dc8e2ec5f26e (127.13.148.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: NON_VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: true } } }
I20260806 04:52:03.921309 20838 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 324249b33cc748309596dc8e2ec5f26e (127.13.148.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: NON_VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: true } } }
I20260806 04:52:03.924882 20867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: NON_VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: true } } }
I20260806 04:52:03.925098 20867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:52:03.926324 20797 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.004s sys 0.005s
I20260806 04:52:03.928117 20978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: NON_VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: true } } }
I20260806 04:52:03.928362 20978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:03.928642 20978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: NON_VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: true } } }
I20260806 04:52:03.928853 20978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:03.933130 20984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:03.933478 20797 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:03.933866 20797 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.006s
I20260806 04:52:03.934000 20797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/wal
uuid: "14c97b2918954c2c90029c4234e84213"
format_stamp: "Formatted at 2026-08-06 04:52:03 on dist-test-slave-1x4g"
server_key: "e24d90fbb9123ec418566d91fcb14697"
server_key_iv: "76cc420d4fd2eefc12a0b9f4952db19b"
server_key_version: "encryptionkey@0"
I20260806 04:52:03.939170 20797 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:03.945844 20797 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 04:52:03.946090 20797 log.cc:1196] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 04:52:03.946736 20797 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: Deleting consensus metadata
I20260806 04:52:03.947016 20797 master_runner.cc:315] Copying system tablet from 127.13.148.190:38833
I20260806 04:52:03.949648 20797 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: tablet copy: Beginning tablet copy session from remote peer at address 127.13.148.190:38833
I20260806 04:52:03.965764 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.950512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50470 (local address 127.13.148.190:38833)
0806 04:52:03.950746 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:03.950756 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:03.951223 (+ 467us) server_negotiation.cc:407] Connection header received
0806 04:52:03.951518 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:03.951523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:03.951590 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:03.951651 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:03.952825 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.953822 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.955745 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.956260 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.956367 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.958859 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:03.958865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:03.958869 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:03.958904 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:03.962664 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.963391 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.963396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.963399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.963596 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:03.964100 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:03.964105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:03.964108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:03.964557 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:03.964663 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:03.965228 (+ 565us) server_negotiation.cc:299] Negotiation successful
0806 04:52:03.965465 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 04:52:03.966768 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:03.950518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:50470)
0806 04:52:03.951115 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:03.951135 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:03.951345 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:03.951827 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:03.951834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:03.952379 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:03.952618 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.952632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.953998 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.954002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.955580 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:03.955598 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:03.956676 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:03.956682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:03.956887 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:03.957589 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:03.957609 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:03.958643 (+ 1034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:03.962797 (+ 4154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.962802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.962806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.963261 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.963733 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:03.963738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:03.963741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:03.963961 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:03.964749 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:03.964753 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:03.964979 (+ 226us) client_negotiation.cc:769] Sending connection context
0806 04:52:03.966168 (+ 1189us) client_negotiation.cc:240] Negotiation successful
0806 04:52:03.966429 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":345,"threads_started":1}
I20260806 04:52:03.967504 20230 tablet_copy_service.cc:140] P 8545f2e2779a47e3ac5cb4398a291b1b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 14c97b2918954c2c90029c4234e84213 ({username='oryx', principal='oryx/127.13.148.188@KRBTEST.COM'} at 127.0.0.1:50470)
I20260806 04:52:03.967649 20230 tablet_copy_service.cc:161] P 8545f2e2779a47e3ac5cb4398a291b1b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 14c97b2918954c2c90029c4234e84213 at {username='oryx', principal='oryx/127.13.148.188@KRBTEST.COM'} at 127.0.0.1:50470: session id = 14c97b2918954c2c90029c4234e84213-00000000000000000000000000000000
I20260806 04:52:03.970940 20230 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:52:03.972757 20797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 04:52:03.979067 20797 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: tablet copy: Starting download of 0 data blocks...
I20260806 04:52:03.979326 20797 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: tablet copy: Starting download of 1 WAL segments...
I20260806 04:52:03.982159 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 20271
I20260806 04:52:03.982152 20797 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:52:03.988982 20230 tablet_copy_service.cc:342] P 8545f2e2779a47e3ac5cb4398a291b1b: Request end of tablet copy session 14c97b2918954c2c90029c4234e84213-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.148.188@KRBTEST.COM'} at 127.0.0.1:50470
I20260806 04:52:03.989117 20230 tablet_copy_service.cc:433] P 8545f2e2779a47e3ac5cb4398a291b1b: ending tablet copy session 14c97b2918954c2c90029c4234e84213-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 14c97b2918954c2c90029c4234e84213
I20260806 04:52:04.001471 20797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:52:04.005357 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:33753
--local_ip_for_outbound_sockets=127.13.148.129
--tserver_master_addrs=127.13.148.190:38833
--webserver_port=36681
--webserver_interface=127.13.148.129
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.129.keytab
--principal=oryx/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:36971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.148.190:38833,127.13.148.189:40159
--tserver_master_addrs=127.13.148.190:38833,127.13.148.189:40159,127.13.148.188:39045 with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:04.007282 20992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:04.008401 20993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:04.016155 20797 server_base.cc:1034] running on GCE node
W20260806 04:52:04.016343 20996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:04.016870 20797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:04.017450 20797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:04.018630 20797 hybrid_clock.cc:648] HybridClock initialized: now 1785991924018601 us; error 25 us; skew 500 ppm
I20260806 04:52:04.018847 20797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:04 dist-test-slave-1x4g krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991924, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:04.021194 20797 init.cc:377] Logged in from keytab as oryx/127.13.148.188@KRBTEST.COM (short username oryx)
I20260806 04:52:04.022341 20797 webserver.cc:460] Webserver started at http://127.13.148.188:39305/ using document root <none> and password file <none>
I20260806 04:52:04.022614 20797 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:04.022703 20797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:04.024689 20797 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260806 04:52:04.026528 21003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:04.026970 20797 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260806 04:52:04.027084 20797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/wal
uuid: "14c97b2918954c2c90029c4234e84213"
format_stamp: "Formatted at 2026-08-06 04:52:03 on dist-test-slave-1x4g"
server_key: "e24d90fbb9123ec418566d91fcb14697"
server_key_iv: "76cc420d4fd2eefc12a0b9f4952db19b"
server_key_version: "encryptionkey@0"
I20260806 04:52:04.027263 20797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:04.043458 20797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:04.046763 20797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:04.109813 20797 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.188:39045
I20260806 04:52:04.109858 21057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.188:39045 every 8 connection(s)
I20260806 04:52:04.110791 20797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/master-2/data/info.pb
I20260806 04:52:04.115448 21058 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:52:04.119119 21058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: Bootstrap starting.
I20260806 04:52:04.136440 21058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:52:04.137014 21058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: Bootstrap complete.
I20260806 04:52:04.138054 21058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: NON_VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: true } }
I20260806 04:52:04.138342 21058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 14c97b2918954c2c90029c4234e84213, State: Initialized, Role: LEARNER
I20260806 04:52:04.138615 21058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: NON_VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: true } }
I20260806 04:52:04.139576 21058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:52:04.139362 21061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: NON_VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: true } } }
I20260806 04:52:04.139853 21061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [sys.catalog]: This master's current role is: LEARNER
I20260806 04:52:04.155798 21072 catalog_manager.cc:1259] Loaded cluster ID: b888a0e01c47497bb126f3d75f810621
I20260806 04:52:04.155895 21072 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: loading cluster ID for follower catalog manager: success
I20260806 04:52:04.159385 21072 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: acquiring CA information for follower catalog manager: success
I20260806 04:52:04.161660 21072 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:52:04.358804 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:04.351988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:39045 (local address 127.0.0.1:54850)
0806 04:52:04.352204 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:04.352230 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:04.352358 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:04.353121 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:04.353127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:04.353160 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:04.353349 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:04.353362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.354605 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.354610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:04.356433 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:04.356458 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.358271 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.358275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:04.358431 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:04.358456 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 04:52:04.358530 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 04:52:04.358616 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 04:52:04.359037 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:04.352170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54850 (local address 127.13.148.188:39045)
0806 04:52:04.352702 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:04.352707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:04.352724 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:52:04.352862 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:04.352868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:04.352970 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:04.353064 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:04.353520 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.354467 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.356695 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.358115 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.358321 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:04.358527 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:04.358580 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 04:52:04.358707 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":291,"threads_started":1}
I20260806 04:52:04.359764 21033 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 04:52:04.360363 21061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8545f2e2779a47e3ac5cb4398a291b1b. Latest consensus state: current_term: 1 leader_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: NON_VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: true } } }
I20260806 04:52:04.360656 21061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [sys.catalog]: This master's current role is: LEARNER
W20260806 04:52:04.364548 20997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:04.364758 20997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:04.364914 20997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:04.372387 20997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:04.372529 20997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:04.372701 20997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:52:04.386278 20997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.129.keytab
--principal=oryx/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:33753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=36681
--tserver_master_addrs=127.13.148.190:38833,127.13.148.189:40159,127.13.148.188:39045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:04.387543 20997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:04.388913 20997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:04.401026 21079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:04.401075 21078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:04.401443 21081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:04.402685 20997 server_base.cc:1034] running on GCE node
I20260806 04:52:04.403241 20997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:04.404099 20997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:04.405283 20997 hybrid_clock.cc:648] HybridClock initialized: now 1785991924405233 us; error 36 us; skew 500 ppm
I20260806 04:52:04.405577 20997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:04 dist-test-slave-1x4g krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991924, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:04.408535 20997 init.cc:377] Logged in from keytab as oryx/127.13.148.129@KRBTEST.COM (short username oryx)
I20260806 04:52:04.409991 20997 webserver.cc:460] Webserver started at http://127.13.148.129:36681/ using document root <none> and password file <none>
I20260806 04:52:04.410912 20997 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:04.411087 20997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:04.415894 20997 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260806 04:52:04.418613 21088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:04.419771 20997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:52:04.419927 20997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "75e962bb879145bd8a56e7cc23142fdf"
format_stamp: "Formatted at 2026-08-06 04:52:01 on dist-test-slave-1x4g"
server_key: "be0deb73a5077aa546238d3837a4c618"
server_key_iv: "4094a7bad04a7699c5328e7ff4db2e82"
server_key_version: "encryptionkey@0"
I20260806 04:52:04.421244 20997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:04.434177 20997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:04.438376 20997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:04.438611 20997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:04.439484 20997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:04.440933 20997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:04.441039 20997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:04.441162 20997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:04.441255 20997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:04.473560 20997 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:33753
I20260806 04:52:04.473629 21203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:33753 every 8 connection(s)
I20260806 04:52:04.474807 20997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:52:04.483582 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 20997
I20260806 04:52:04.483939 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 20409
Aug 06 04:52:04 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991924, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.188@KRBTEST.COM
Aug 06 04:52:04 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991924, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.189@KRBTEST.COM
Aug 06 04:52:04 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991924, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM
Aug 06 04:52:04 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.130@KRBTEST.COM
I20260806 04:52:04.519356 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:04.477350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:37735 (local address 127.13.148.188:39045)
0806 04:52:04.477550 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:04.477556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:04.489503 (+ 11947us) server_negotiation.cc:407] Connection header received
0806 04:52:04.498074 (+ 8571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:04.498082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:04.498200 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:04.498295 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:04.503085 (+ 4790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.504339 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.506825 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.507388 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.507471 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:04.512196 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:04.512202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:04.512206 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:04.512245 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:04.515768 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:04.516614 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:04.516619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:04.516623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:04.516811 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:04.517336 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:04.517341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:04.517345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:04.517715 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:04.517886 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:04.518799 (+ 913us) server_negotiation.cc:299] Negotiation successful
0806 04:52:04.519043 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 04:52:04.520394 21211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:04.478304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:48671 (local address 127.13.148.189:40159)
0806 04:52:04.501483 (+ 23179us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:04.501489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:04.502037 (+ 548us) server_negotiation.cc:407] Connection header received
0806 04:52:04.502444 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:04.502449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:04.502540 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:04.502638 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:04.504362 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.505425 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.507151 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.507667 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.507768 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:04.513004 (+ 5236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:04.513011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:04.513015 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:04.513092 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:04.516678 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:04.517546 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:04.517551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:04.517554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:04.517688 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:04.518134 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:04.518138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:04.518141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:04.518532 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:04.518676 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:04.519853 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0806 04:52:04.520013 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23036,"thread_start_us":22907,"threads_started":1}
I20260806 04:52:04.522145 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:04.477392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:42871 (local address 127.13.148.190:38833)
0806 04:52:04.502523 (+ 25131us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:04.502529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:04.502547 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:52:04.502601 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:04.502605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:04.502673 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:04.504395 (+ 1722us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:04.504596 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.505672 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.507461 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.508012 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.508722 (+ 710us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:04.514571 (+ 5849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:04.514577 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:04.514581 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:04.514617 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:04.518229 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:04.519382 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:04.519387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:04.519391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:04.519530 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:04.519996 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:04.520003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:04.520006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:04.520411 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:04.521466 (+ 1055us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:04.521597 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 04:52:04.521820 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25003,"thread_start_us":25014,"threads_started":1}
I20260806 04:52:04.522221 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:04.478004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.129:42871)
0806 04:52:04.488737 (+ 10733us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:04.492926 (+ 4189us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:04.496821 (+ 3895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:04.502832 (+ 6011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:04.502836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:04.503744 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:04.504244 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:04.504259 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.505822 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.505830 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:04.507308 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:04.507321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.508226 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.508233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:04.508443 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:04.509909 (+ 1466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:04.509936 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:04.514328 (+ 4392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:04.518427 (+ 4099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:04.518437 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:04.518442 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:04.519204 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:04.519654 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:04.519658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:04.519661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:04.519840 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:04.520562 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:04.520566 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:04.520722 (+ 156us) client_negotiation.cc:769] Sending connection context
0806 04:52:04.520886 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 04:52:04.521168 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10367,"mutex_wait_us":4,"thread_start_us":22648,"threads_started":1}
I20260806 04:52:04.522614 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:04.478004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:40159 (local address 127.13.148.129:48671)
0806 04:52:04.501948 (+ 23944us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:04.501961 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:04.502320 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:04.502883 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:04.502887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:04.503735 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:04.504205 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:04.504220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.505584 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.505589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:04.507011 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:04.507022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.507999 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.508004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:04.508138 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:04.508927 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:04.508954 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:04.512785 (+ 3831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:04.516844 (+ 4059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:04.516852 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:04.516858 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:04.517412 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:04.517838 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:04.517860 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:04.517863 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:04.518019 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:04.518735 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:04.518739 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:04.518896 (+ 157us) client_negotiation.cc:769] Sending connection context
0806 04:52:04.519070 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 04:52:04.520836 (+ 1766us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22907,"mutex_wait_us":503,"thread_start_us":14293,"threads_started":1}
I20260806 04:52:04.522934 21212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:04.478004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:39045 (local address 127.13.148.129:37735)
0806 04:52:04.489300 (+ 11296us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:04.489345 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:04.496473 (+ 7128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:04.498819 (+ 2346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:04.498833 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:04.499733 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:04.502883 (+ 3150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:04.502905 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.504570 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.504576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:04.506691 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:04.506703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.507765 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.507772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:04.508018 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:04.508927 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:04.508961 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:04.511985 (+ 3024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:04.515924 (+ 3939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:04.515932 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:04.515946 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:04.516469 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:04.516966 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:04.516972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:04.516978 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:04.517199 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:04.517979 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:04.517985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:04.518397 (+ 412us) client_negotiation.cc:769] Sending connection context
0806 04:52:04.519619 (+ 1222us) client_negotiation.cc:240] Negotiation successful
0806 04:52:04.520905 (+ 1286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11123,"mutex_wait_us":1128,"spinlock_wait_cycles":46592,"thread_start_us":16396,"threads_started":1}
I20260806 04:52:04.523847 21205 heartbeater.cc:347] Connected to a master server at 127.13.148.190:38833
I20260806 04:52:04.524041 21204 heartbeater.cc:347] Connected to a master server at 127.13.148.188:39045
I20260806 04:52:04.524183 21204 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:04.524186 21205 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:04.524264 21206 heartbeater.cc:347] Connected to a master server at 127.13.148.189:40159
I20260806 04:52:04.524484 21206 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:04.524494 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:04.485423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.13.148.130:46089)
0806 04:52:04.501622 (+ 16199us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:04.501632 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:04.502342 (+ 710us) server_negotiation.cc:407] Connection header received
0806 04:52:04.502602 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:04.502611 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:04.502839 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:04.503035 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:04.505248 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.506606 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.508245 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:04.509424 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:04.509565 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:04.516384 (+ 6819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:04.516397 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:04.516413 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:04.516477 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:04.520560 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:04.521422 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:04.521430 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:04.521433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:04.521569 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:04.522005 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:04.522009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:04.522013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:04.522432 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:04.522712 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:04.523792 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0806 04:52:04.524068 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16009,"thread_start_us":15826,"threads_started":1}
I20260806 04:52:04.525204 21204 heartbeater.cc:511] Master 127.13.148.188:39045 requested a full tablet report, sending...
I20260806 04:52:04.525230 21206 heartbeater.cc:511] Master 127.13.148.189:40159 requested a full tablet report, sending...
I20260806 04:52:04.525887 21205 heartbeater.cc:511] Master 127.13.148.190:38833 requested a full tablet report, sending...
I20260806 04:52:04.526003 20536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:04.526904 20210 ts_manager.cc:194] Re-registered known tserver with Master: 75e962bb879145bd8a56e7cc23142fdf (127.13.148.129:33753)
I20260806 04:52:04.527529 20829 ts_manager.cc:194] Registered new tserver with Master: 75e962bb879145bd8a56e7cc23142fdf (127.13.148.129:33753)
I20260806 04:52:04.527530 21022 ts_manager.cc:194] Registered new tserver with Master: 75e962bb879145bd8a56e7cc23142fdf (127.13.148.129:33753)
I20260806 04:52:04.527774 20210 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:42871
I20260806 04:52:04.605919 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 20409
I20260806 04:52:04.623293 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:46089
--local_ip_for_outbound_sockets=127.13.148.130
--tserver_master_addrs=127.13.148.190:38833
--webserver_port=32853
--webserver_interface=127.13.148.130
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.130.keytab
--principal=oryx/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:36971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.148.190:38833,127.13.148.189:40159
--tserver_master_addrs=127.13.148.190:38833,127.13.148.189:40159,127.13.148.188:39045 with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:52:04.870260 20977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: NON_VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: true } } }
I20260806 04:52:04.870668 20977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:04.872486 21217 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b: attempting to promote NON_VOTER 14c97b2918954c2c90029c4234e84213 to VOTER
I20260806 04:52:04.873062 21217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } }
I20260806 04:52:04.874692 20838 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 FOLLOWER]: Refusing update from remote peer 8545f2e2779a47e3ac5cb4398a291b1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 04:52:04.874720 21033 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 1 LEARNER]: Refusing update from remote peer 8545f2e2779a47e3ac5cb4398a291b1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 04:52:04.875257 20977 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 04:52:04.875658 21217 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 04:52:04.878302 20977 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 14c97b2918954c2c90029c4234e84213 (127.13.148.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } } }
I20260806 04:52:04.882193 20838 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 14c97b2918954c2c90029c4234e84213 (127.13.148.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } } }
I20260806 04:52:04.882704 20977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } } }
I20260806 04:52:04.883013 20977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:04.882651 21033 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 14c97b2918954c2c90029c4234e84213 (127.13.148.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } } }
I20260806 04:52:04.883288 20977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } } }
I20260806 04:52:04.883528 20977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:04.883724 20977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } } }
I20260806 04:52:04.883945 20977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:04.887502 21222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } } }
I20260806 04:52:04.887818 21222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:52:04.889156 21221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } } }
I20260806 04:52:04.889504 21221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: This master's current role is: FOLLOWER
W20260806 04:52:04.929513 21216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:04.929792 21216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:04.929935 21216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:04.936344 21216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:04.936425 21216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:04.936515 21216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:52:04.945016 21216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.130.keytab
--principal=oryx/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:46089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=32853
--tserver_master_addrs=127.13.148.190:38833,127.13.148.189:40159,127.13.148.188:39045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:04.946117 21216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:04.947259 21216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:04.959555 21228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:04.959527 21227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:04.959947 21230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:04.960156 21216 server_base.cc:1034] running on GCE node
I20260806 04:52:04.960806 21216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:04.961730 21216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:04.962915 21216 hybrid_clock.cc:648] HybridClock initialized: now 1785991924962878 us; error 27 us; skew 500 ppm
I20260806 04:52:04.963189 21216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:04 dist-test-slave-1x4g krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991924, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:04.965957 21216 init.cc:377] Logged in from keytab as oryx/127.13.148.130@KRBTEST.COM (short username oryx)
I20260806 04:52:04.967116 21216 webserver.cc:460] Webserver started at http://127.13.148.130:32853/ using document root <none> and password file <none>
I20260806 04:52:04.967681 21216 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:04.967813 21216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:04.972869 21216 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:52:04.975250 21238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:04.976459 21216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 04:52:04.976615 21216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "06e4f08b79e043fcae6f7a6ad7b8f281"
format_stamp: "Formatted at 2026-08-06 04:52:02 on dist-test-slave-1x4g"
server_key: "61047ff539180177448e4c55bbfe1214"
server_key_iv: "23f1d2b7b83958600ce7caee00e10006"
server_key_version: "encryptionkey@0"
I20260806 04:52:04.977595 21216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:04.992470 21216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:04.995795 21216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:04.996042 21216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:04.996963 21216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:04.998287 21216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:04.998399 21216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:04.998510 21216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:04.998589 21216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:05.029758 21216 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:46089
I20260806 04:52:05.029829 21351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:46089 every 8 connection(s)
I20260806 04:52:05.031229 21216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:52:05.032979 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 21216
I20260806 04:52:05.033257 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 20546
Aug 06 04:52:05 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991924, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.189@KRBTEST.COM
Aug 06 04:52:05 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991924, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.188@KRBTEST.COM
Aug 06 04:52:05 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991924, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM
Aug 06 04:52:05 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.131@KRBTEST.COM
I20260806 04:52:05.069974 21360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.035131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:38123 (local address 127.13.148.188:39045)
0806 04:52:05.044249 (+ 9118us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:05.044257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:05.051049 (+ 6792us) server_negotiation.cc:407] Connection header received
0806 04:52:05.051200 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:05.051205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:05.051277 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:05.051382 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:05.053675 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.054661 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.057555 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.058200 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.058284 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.063308 (+ 5024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:05.063314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:05.063317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:05.063351 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:05.066466 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.067281 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.067286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.067288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.067410 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.067895 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.067901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.067904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.068243 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:05.068374 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:05.069347 (+ 973us) server_negotiation.cc:299] Negotiation successful
0806 04:52:05.069550 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8960,"thread_start_us":8835,"threads_started":1}
I20260806 04:52:05.072197 21363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.034556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:40159 (local address 127.13.148.130:36571)
0806 04:52:05.045731 (+ 11175us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:05.045794 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:05.050342 (+ 4548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:05.051576 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:05.051580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:05.052512 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:05.053504 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.053528 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.054925 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.054932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.057359 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.057375 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.058831 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.058836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.059077 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.059866 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:05.059887 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:05.063788 (+ 3901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:05.069335 (+ 5547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:05.069342 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:05.069347 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:05.069800 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:05.070217 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:05.070222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:05.070225 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:05.070463 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:05.071210 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:05.071216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:05.071368 (+ 152us) client_negotiation.cc:769] Sending connection context
0806 04:52:05.071537 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 04:52:05.071768 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10839,"mutex_wait_us":312,"spinlock_wait_cycles":11520,"thread_start_us":16482,"threads_started":1}
I20260806 04:52:05.072338 21358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.034556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:39045 (local address 127.13.148.130:38123)
0806 04:52:05.050961 (+ 16405us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:05.050977 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:05.051088 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:05.051417 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:05.051421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:05.052508 (+ 1087us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:05.053507 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.053528 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.054868 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.054872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.057378 (+ 2506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.057395 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.058595 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.058603 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.058935 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.059598 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:05.059628 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:05.063107 (+ 3479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:05.066600 (+ 3493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:05.066608 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:05.066621 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:05.067163 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:05.067528 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:05.067532 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:05.067535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:05.067768 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:05.068472 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:05.068480 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:05.068880 (+ 400us) client_negotiation.cc:769] Sending connection context
0806 04:52:05.070201 (+ 1321us) client_negotiation.cc:240] Negotiation successful
0806 04:52:05.070729 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16271,"mutex_wait_us":3239,"spinlock_wait_cycles":15616,"thread_start_us":8169,"threads_started":1}
I20260806 04:52:05.072223 21361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.035112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.130:47097)
0806 04:52:05.050082 (+ 14970us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:05.050098 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:05.050342 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:05.051363 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:05.051374 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:05.052512 (+ 1138us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:05.053511 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.053528 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.054869 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.054874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.057378 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.057387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.058594 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.058602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.058941 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.059795 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:05.059826 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:05.063392 (+ 3566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:05.068492 (+ 5100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:05.068496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:05.068499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:05.069009 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:05.069430 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:05.069436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:05.069442 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:05.069596 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:05.070686 (+ 1090us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:05.070693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:05.070847 (+ 154us) client_negotiation.cc:769] Sending connection context
0806 04:52:05.071011 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 04:52:05.071280 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14890,"mutex_wait_us":562,"thread_start_us":14642,"threads_started":1}
I20260806 04:52:05.073621 21357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.034463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58476 (local address 127.13.148.131:46529)
0806 04:52:05.035186 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:05.035194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:05.036002 (+ 808us) server_negotiation.cc:407] Connection header received
0806 04:52:05.036203 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:05.036211 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:05.036386 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:05.036521 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:05.038855 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.041092 (+ 2237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.043328 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.044307 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.044464 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.065770 (+ 21306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:05.065789 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:05.065806 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:05.065863 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:05.069561 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.070519 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.070527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.070531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.070696 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.071217 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.071222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.071226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.071674 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:05.071935 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:05.073118 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0806 04:52:05.073365 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":1065,"threads_started":1}
I20260806 04:52:05.073700 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.035447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:36571 (local address 127.13.148.189:40159)
0806 04:52:05.050000 (+ 14553us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:05.050005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:05.050022 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:52:05.050735 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:05.050741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:05.050833 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:05.050938 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:05.053686 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.054782 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.057535 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.058506 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.058607 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.063972 (+ 5365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:05.063977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:05.063982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:05.064031 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:05.069204 (+ 5173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.069944 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.069949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.069952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.070090 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.070605 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.070610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.070612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.071012 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:05.071150 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:05.073177 (+ 2027us) server_negotiation.cc:299] Negotiation successful
0806 04:52:05.073405 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14440,"thread_start_us":14337,"threads_started":1}
I20260806 04:52:05.073813 21352 heartbeater.cc:347] Connected to a master server at 127.13.148.188:39045
I20260806 04:52:05.074088 21352 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:05.074834 20673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:05.074868 21352 heartbeater.cc:511] Master 127.13.148.188:39045 requested a full tablet report, sending...
I20260806 04:52:05.076776 21354 heartbeater.cc:347] Connected to a master server at 127.13.148.189:40159
I20260806 04:52:05.076848 21022 ts_manager.cc:194] Registered new tserver with Master: 06e4f08b79e043fcae6f7a6ad7b8f281 (127.13.148.130:46089)
I20260806 04:52:05.076959 21354 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:05.077538 21354 heartbeater.cc:511] Master 127.13.148.189:40159 requested a full tablet report, sending...
I20260806 04:52:05.078790 20829 ts_manager.cc:194] Registered new tserver with Master: 06e4f08b79e043fcae6f7a6ad7b8f281 (127.13.148.130:46089)
I20260806 04:52:05.079066 21359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.035091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:47097 (local address 127.13.148.190:38833)
0806 04:52:05.035564 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:05.035569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:05.050185 (+ 14616us) server_negotiation.cc:407] Connection header received
0806 04:52:05.050755 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:05.050761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:05.050833 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:05.050946 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:05.053686 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.054708 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.057535 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.058205 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.058313 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.063583 (+ 5270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:05.063589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:05.063594 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:05.063651 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:05.068339 (+ 4688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.069152 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.069156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.069159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.069314 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.069724 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.069729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.069734 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.070479 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:05.070613 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:05.072256 (+ 1643us) server_negotiation.cc:299] Negotiation successful
0806 04:52:05.077100 (+ 4844us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":566,"threads_started":1}
I20260806 04:52:05.079720 21353 heartbeater.cc:347] Connected to a master server at 127.13.148.190:38833
I20260806 04:52:05.079842 21353 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:05.080204 21353 heartbeater.cc:511] Master 127.13.148.190:38833 requested a full tablet report, sending...
I20260806 04:52:05.081103 20210 ts_manager.cc:194] Re-registered known tserver with Master: 06e4f08b79e043fcae6f7a6ad7b8f281 (127.13.148.130:46089)
I20260806 04:52:05.082118 20210 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:47097
I20260806 04:52:05.152877 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 20546
I20260806 04:52:05.169894 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:46529
--local_ip_for_outbound_sockets=127.13.148.131
--tserver_master_addrs=127.13.148.190:38833
--webserver_port=40339
--webserver_interface=127.13.148.131
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.131.keytab
--principal=oryx/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:36971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.148.190:38833,127.13.148.189:40159
--tserver_master_addrs=127.13.148.190:38833,127.13.148.189:40159,127.13.148.188:39045 with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:05.471835 21365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:05.472092 21365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:05.472230 21365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:05.478441 21365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:05.478564 21365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:05.478703 21365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:52:05.487635 21365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.131.keytab
--principal=oryx/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:46529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=40339
--tserver_master_addrs=127.13.148.190:38833,127.13.148.189:40159,127.13.148.188:39045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:05.488687 21365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:05.489818 21365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:05.502276 21371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:05.502259 21374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:05.502305 21372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:05.503003 21365 server_base.cc:1034] running on GCE node
I20260806 04:52:05.503502 21365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:05.504261 21365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:05.505525 21365 hybrid_clock.cc:648] HybridClock initialized: now 1785991925505475 us; error 48 us; skew 500 ppm
I20260806 04:52:05.505864 21365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:05 dist-test-slave-1x4g krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991925, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:05.508713 21365 init.cc:377] Logged in from keytab as oryx/127.13.148.131@KRBTEST.COM (short username oryx)
I20260806 04:52:05.509920 21365 webserver.cc:460] Webserver started at http://127.13.148.131:40339/ using document root <none> and password file <none>
I20260806 04:52:05.510552 21365 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:05.510664 21365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:05.515486 21365 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:52:05.518018 21381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:05.519157 21365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:52:05.519276 21365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "b39b7889a4cd4aa781f52d643dec6432"
format_stamp: "Formatted at 2026-08-06 04:52:02 on dist-test-slave-1x4g"
server_key: "857f15d9d36934225832b3d5dc110503"
server_key_iv: "d02c46a5795cf344c1339ead7f7adb32"
server_key_version: "encryptionkey@0"
I20260806 04:52:05.520076 21365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:05.530112 21205 heartbeater.cc:503] Master 127.13.148.190:38833 was elected leader, sending a full tablet report...
I20260806 04:52:05.540764 21365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:05.544391 21365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:05.544684 21365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:05.545583 21365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:05.546748 21365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:05.546856 21365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:05.546969 21365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:05.547050 21365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:05.576843 21365 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:46529
I20260806 04:52:05.576885 21495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:46529 every 8 connection(s)
I20260806 04:52:05.578229 21365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:52:05.579073 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 21365
Aug 06 04:52:05 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991925, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.188@KRBTEST.COM
Aug 06 04:52:05 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991925, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM
I20260806 04:52:05.656229 21501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.581233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:43817 (local address 127.13.148.188:39045)
0806 04:52:05.593689 (+ 12456us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:05.593699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:05.600191 (+ 6492us) server_negotiation.cc:407] Connection header received
0806 04:52:05.603849 (+ 3658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:05.603858 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:05.603948 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:05.604022 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:05.611599 (+ 7577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.612736 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.617230 (+ 4494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.618112 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.618225 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.643314 (+ 25089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:05.643322 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:05.643330 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:05.643378 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:05.646931 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.650507 (+ 3576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.650516 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.650522 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.650717 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.652038 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.652046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.652051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.652446 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:05.652651 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:05.654231 (+ 1580us) server_negotiation.cc:299] Negotiation successful
0806 04:52:05.654489 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12285,"thread_start_us":12472,"threads_started":1}
I20260806 04:52:05.659211 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.581141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:46357 (local address 127.13.148.190:38833)
0806 04:52:05.596868 (+ 15727us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:05.596877 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:05.599975 (+ 3098us) server_negotiation.cc:407] Connection header received
0806 04:52:05.603707 (+ 3732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:05.603712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:05.603801 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:05.603897 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:05.606992 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.608148 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.611253 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.611958 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.612066 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.641099 (+ 29033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:05.641109 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:05.641116 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:05.641167 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:05.645421 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.648192 (+ 2771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.648199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.648205 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.648409 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.654444 (+ 6035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.654449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.654452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.654910 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:05.655125 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:05.658372 (+ 3247us) server_negotiation.cc:299] Negotiation successful
0806 04:52:05.658611 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15550,"thread_start_us":15433,"threads_started":1}
I20260806 04:52:05.659248 21504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.582100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.131:46357)
0806 04:52:05.599767 (+ 17667us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:05.599817 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:05.603565 (+ 3748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:05.604534 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:05.604550 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:05.605741 (+ 1191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:05.606788 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.606817 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.608287 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.608295 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.611077 (+ 2782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.611113 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.620068 (+ 8955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.620076 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.620307 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.621651 (+ 1344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:05.621687 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:05.640887 (+ 19200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:05.647145 (+ 6258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:05.647157 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:05.647179 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:05.648065 (+ 886us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:05.652728 (+ 4663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:05.652735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:05.652741 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:05.652957 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:05.657926 (+ 4969us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:05.657931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:05.658084 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 04:52:05.658232 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 04:52:05.658520 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17315,"thread_start_us":15472,"threads_started":1}
I20260806 04:52:05.661206 21497 heartbeater.cc:347] Connected to a master server at 127.13.148.190:38833
I20260806 04:52:05.661635 21497 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:05.662628 21497 heartbeater.cc:511] Master 127.13.148.190:38833 requested a full tablet report, sending...
I20260806 04:52:05.660471 21502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.582713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:39045 (local address 127.13.148.131:43817)
0806 04:52:05.600093 (+ 17380us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:05.600106 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:05.602831 (+ 2725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:05.607035 (+ 4204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:05.607040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:05.607462 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:05.607737 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.607748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.612986 (+ 5238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.612994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.614904 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.614921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.618655 (+ 3734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.618663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.619020 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.622865 (+ 3845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:05.622905 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:05.638020 (+ 15115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:05.648221 (+ 10201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:05.648225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:05.648228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:05.650081 (+ 1853us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:05.650938 (+ 857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:05.650946 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:05.650950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:05.651278 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:05.653179 (+ 1901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:05.653191 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:05.653702 (+ 511us) client_negotiation.cc:769] Sending connection context
0806 04:52:05.656900 (+ 3198us) client_negotiation.cc:240] Negotiation successful
0806 04:52:05.657640 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17262,"spinlock_wait_cycles":69504,"thread_start_us":15190,"threads_started":1}
I20260806 04:52:05.664359 20210 ts_manager.cc:194] Re-registered known tserver with Master: b39b7889a4cd4aa781f52d643dec6432 (127.13.148.131:46529)
I20260806 04:52:05.664940 21496 heartbeater.cc:347] Connected to a master server at 127.13.148.188:39045
I20260806 04:52:05.665055 21496 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:05.665418 21496 heartbeater.cc:511] Master 127.13.148.188:39045 requested a full tablet report, sending...
I20260806 04:52:05.665427 20210 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:46357
I20260806 04:52:05.666535 21022 ts_manager.cc:194] Registered new tserver with Master: b39b7889a4cd4aa781f52d643dec6432 (127.13.148.131:46529)
I20260806 04:52:05.669039 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.602604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54864 (local address 127.13.148.188:39045)
0806 04:52:05.636428 (+ 33824us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:05.636436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:05.643712 (+ 7276us) server_negotiation.cc:407] Connection header received
0806 04:52:05.648271 (+ 4559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:05.648279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:05.648389 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:05.648511 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:05.652505 (+ 3994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.653657 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.657507 (+ 3850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.658178 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.658257 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.661997 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:05.662006 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:05.662012 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:05.662048 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:05.665712 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.666560 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.666565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.666568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.666714 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.667108 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.667113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.667115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.667417 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:05.667646 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:05.668478 (+ 832us) server_negotiation.cc:299] Negotiation successful
0806 04:52:05.668700 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33675,"thread_start_us":33827,"threads_started":1}
Aug 06 04:52:05 dist-test-slave-1x4g krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991925, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.189@KRBTEST.COM
I20260806 04:52:05.673981 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.602213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.13.148.190:38833)
0806 04:52:05.637598 (+ 35385us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:05.637607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:05.638627 (+ 1020us) server_negotiation.cc:407] Connection header received
0806 04:52:05.638699 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:05.638706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:05.638803 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:05.638922 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:05.644154 (+ 5232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.645668 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.652425 (+ 6757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.653131 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.653240 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.662458 (+ 9218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:05.662463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:05.662468 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:05.662503 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:05.670425 (+ 7922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.671263 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.671270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.671274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.671453 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.671923 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.671930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.671935 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.672330 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:05.672568 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:05.673352 (+ 784us) server_negotiation.cc:299] Negotiation successful
0806 04:52:05.673612 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35194,"thread_start_us":35435,"threads_started":1}
I20260806 04:52:05.680128 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.603309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:47645 (local address 127.13.148.189:40159)
0806 04:52:05.651859 (+ 48550us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:05.651867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:05.654003 (+ 2136us) server_negotiation.cc:407] Connection header received
0806 04:52:05.654082 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:05.654089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:05.654188 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:05.654289 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:05.656042 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.657299 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.660724 (+ 3425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.661501 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.661623 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.672878 (+ 11255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:05.672884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:05.672888 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:05.672924 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:05.676258 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.677049 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.677056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.677062 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.677962 (+ 900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.678684 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.678688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.678691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.679074 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:05.679214 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:05.679587 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 04:52:05.679779 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48406,"thread_start_us":59817,"threads_started":1}
I20260806 04:52:05.680274 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.601165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:40159 (local address 127.13.148.131:47645)
0806 04:52:05.652376 (+ 51211us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:05.653377 (+ 1001us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:05.653484 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:05.654321 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:05.654325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:05.654882 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:05.655161 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.655174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.658966 (+ 3792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.658971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.660535 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.660552 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.668829 (+ 8277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.668841 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.669020 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.670126 (+ 1106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:05.670147 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:05.672680 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:05.676391 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:05.676398 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:05.676403 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:05.676931 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:05.678333 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:05.678336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:05.678338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:05.678565 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:05.679199 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:05.679203 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:05.679359 (+ 156us) client_negotiation.cc:769] Sending connection context
0806 04:52:05.679496 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 04:52:05.679822 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51100,"thread_start_us":50958,"threads_started":1}
I20260806 04:52:05.680806 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.627896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47822 (local address 127.13.148.189:40159)
0806 04:52:05.662304 (+ 34408us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:05.662311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:05.663789 (+ 1478us) server_negotiation.cc:407] Connection header received
0806 04:52:05.664047 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:05.664054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:05.664137 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:05.664217 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:05.666335 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.667464 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.669414 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.669947 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.670072 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.673474 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:05.673481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:05.673486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:05.673522 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:05.677327 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.678080 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.678087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.678090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.678224 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:05.678668 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:05.678673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:05.678676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:05.678938 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:05.679158 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:05.680095 (+ 937us) server_negotiation.cc:299] Negotiation successful
0806 04:52:05.680343 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34277,"thread_start_us":23967,"threads_started":1}
I20260806 04:52:05.681571 21498 heartbeater.cc:347] Connected to a master server at 127.13.148.189:40159
I20260806 04:52:05.681716 21498 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:05.682103 21498 heartbeater.cc:511] Master 127.13.148.189:40159 requested a full tablet report, sending...
I20260806 04:52:05.683056 20828 ts_manager.cc:194] Registered new tserver with Master: b39b7889a4cd4aa781f52d643dec6432 (127.13.148.131:46529)
I20260806 04:52:05.678748 20210 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:52:05.687553 20210 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:52:05.723358 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.714463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:46529 (local address 127.0.0.1:58492)
0806 04:52:05.715005 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:05.715025 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:05.715312 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:05.717498 (+ 2186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:05.717503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:05.717526 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:05.717924 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.717933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.719221 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.719226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.720646 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.720658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.722639 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.722644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.722828 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.722852 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 04:52:05.722941 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 04:52:05.723052 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":291,"threads_started":1}
I20260806 04:52:05.723495 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.714787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58492 (local address 127.13.148.131:46529)
0806 04:52:05.717042 (+ 2255us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:05.717047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:05.717073 (+ 26us) server_negotiation.cc:407] Connection header received
0806 04:52:05.717142 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:05.717147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:05.717318 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:05.717499 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:05.718108 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.719090 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.720815 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.722447 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.722609 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.722767 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:05.722988 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 04:52:05.723150 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2134,"thread_start_us":1475,"threads_started":1}
I20260806 04:52:05.727053 21430 tablet_service.cc:1467] Processing CreateTablet for tablet 02416b6a1e5e4feb97e851ceda26aacd (DEFAULT_TABLE table=test-table [id=c3b34ddfe02c487fbf65c0c6eb0cd78a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:05.728438 21430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02416b6a1e5e4feb97e851ceda26aacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:05.728615 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.716074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.13.148.130:46089)
0806 04:52:05.716511 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:05.716517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:05.721498 (+ 4981us) server_negotiation.cc:407] Connection header received
0806 04:52:05.721711 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:05.721723 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:05.721960 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:05.722145 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:05.722948 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.724163 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.726408 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.727638 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.727887 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.728046 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:05.728223 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 04:52:05.728360 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":539,"threads_started":1}
I20260806 04:52:05.728634 21527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.715138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:46089 (local address 127.0.0.1:48834)
0806 04:52:05.721341 (+ 6203us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:05.721363 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:05.721505 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:05.722478 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:05.722482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:05.722506 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:05.722810 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.722817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.724274 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.724279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.726249 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.726265 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.727905 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.727910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.728068 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.728099 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 04:52:05.728186 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 04:52:05.728291 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6059,"mutex_wait_us":1,"thread_start_us":5866,"threads_started":1}
I20260806 04:52:05.732357 21287 tablet_service.cc:1467] Processing CreateTablet for tablet 02416b6a1e5e4feb97e851ceda26aacd (DEFAULT_TABLE table=test-table [id=c3b34ddfe02c487fbf65c0c6eb0cd78a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:05.733722 21287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02416b6a1e5e4feb97e851ceda26aacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:05.734858 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.715257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:33753 (local address 127.0.0.1:33282)
0806 04:52:05.721986 (+ 6729us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:05.722002 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:05.722175 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:05.724124 (+ 1949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:05.724127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:05.724148 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:05.724548 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.724563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.725827 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.725830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.727411 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.727426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.734142 (+ 6716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.734146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.734252 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.734271 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:52:05.734360 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 04:52:05.734498 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6619,"thread_start_us":6477,"threads_started":1}
I20260806 04:52:05.734850 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.715706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.13.148.129:33753)
0806 04:52:05.716145 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:05.716151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:05.722084 (+ 5933us) server_negotiation.cc:407] Connection header received
0806 04:52:05.722353 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:05.722361 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:05.723743 (+ 1382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:05.723901 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:05.724765 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.725663 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.727608 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.733581 (+ 5973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.733933 (+ 352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.734045 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:05.734425 (+ 380us) server_negotiation.cc:299] Negotiation successful
0806 04:52:05.734572 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":749,"threads_started":1}
I20260806 04:52:05.737602 21137 tablet_service.cc:1467] Processing CreateTablet for tablet 02416b6a1e5e4feb97e851ceda26aacd (DEFAULT_TABLE table=test-table [id=c3b34ddfe02c487fbf65c0c6eb0cd78a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:05.738732 21137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02416b6a1e5e4feb97e851ceda26aacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:05.744608 21531 tablet_bootstrap.cc:492] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432: Bootstrap starting.
I20260806 04:52:05.747823 21532 tablet_bootstrap.cc:492] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281: Bootstrap starting.
I20260806 04:52:05.748728 21531 tablet_bootstrap.cc:654] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:05.749929 21531 log.cc:824] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432: Log is configured to *not* fsync() on all Append() calls
I20260806 04:52:05.750687 21532 tablet_bootstrap.cc:654] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:05.750967 21533 tablet_bootstrap.cc:492] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf: Bootstrap starting.
I20260806 04:52:05.752775 21532 log.cc:824] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281: Log is configured to *not* fsync() on all Append() calls
I20260806 04:52:05.754216 21533 tablet_bootstrap.cc:654] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:05.755333 21531 tablet_bootstrap.cc:492] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432: No bootstrap required, opened a new log
I20260806 04:52:05.755451 21533 log.cc:824] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf: Log is configured to *not* fsync() on all Append() calls
I20260806 04:52:05.755618 21531 ts_tablet_manager.cc:1397] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260806 04:52:05.756110 21532 tablet_bootstrap.cc:492] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281: No bootstrap required, opened a new log
I20260806 04:52:05.756374 21532 ts_tablet_manager.cc:1397] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260806 04:52:05.758282 21533 tablet_bootstrap.cc:492] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf: No bootstrap required, opened a new log
I20260806 04:52:05.758601 21533 ts_tablet_manager.cc:1397] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260806 04:52:05.763626 21532 raft_consensus.cc:348] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } } peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } }
I20260806 04:52:05.763967 21532 raft_consensus.cc:374] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:05.764083 21532 raft_consensus.cc:729] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e4f08b79e043fcae6f7a6ad7b8f281, State: Initialized, Role: FOLLOWER
I20260806 04:52:05.764590 21532 consensus_queue.cc:260] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } } peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } }
I20260806 04:52:05.765671 21531 raft_consensus.cc:348] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } } peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } }
I20260806 04:52:05.766023 21531 raft_consensus.cc:374] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:05.766072 21532 ts_tablet_manager.cc:1428] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260806 04:52:05.766103 21531 raft_consensus.cc:729] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b39b7889a4cd4aa781f52d643dec6432, State: Initialized, Role: FOLLOWER
I20260806 04:52:05.766420 21353 heartbeater.cc:503] Master 127.13.148.190:38833 was elected leader, sending a full tablet report...
I20260806 04:52:05.768839 21531 consensus_queue.cc:260] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } } peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } }
I20260806 04:52:05.769534 21533 raft_consensus.cc:348] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } } peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } }
I20260806 04:52:05.769979 21533 raft_consensus.cc:374] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:05.770112 21533 raft_consensus.cc:729] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e962bb879145bd8a56e7cc23142fdf, State: Initialized, Role: FOLLOWER
I20260806 04:52:05.770711 21533 consensus_queue.cc:260] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } } peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } }
I20260806 04:52:05.771319 21497 heartbeater.cc:503] Master 127.13.148.190:38833 was elected leader, sending a full tablet report...
I20260806 04:52:05.771929 21531 ts_tablet_manager.cc:1428] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260806 04:52:05.773291 21533 ts_tablet_manager.cc:1428] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf: Time spent starting tablet: real 0.015s user 0.007s sys 0.006s
W20260806 04:52:05.784168 21355 tablet.cc:2369] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:52:05.850186 21506 tablet.cc:2369] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:52:05.915364 21539 raft_consensus.cc:482] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:52:05.915710 21539 raft_consensus.cc:504] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } } peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } }
I20260806 04:52:05.917686 21539 leader_election.cc:288] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b39b7889a4cd4aa781f52d643dec6432 (127.13.148.131:46529), 06e4f08b79e043fcae6f7a6ad7b8f281 (127.13.148.130:46089)
I20260806 04:52:05.927181 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.918118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:41927 (local address 127.13.148.131:46529)
0806 04:52:05.918436 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:05.918442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:05.919009 (+ 567us) server_negotiation.cc:407] Connection header received
0806 04:52:05.919219 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:05.919224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:05.919299 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:05.919396 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:05.920269 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.921558 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.924861 (+ 3303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.926079 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.926254 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.926328 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:05.926810 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 04:52:05.926934 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:52:05.927177 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.918118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:40931 (local address 127.13.148.130:46089)
0806 04:52:05.918372 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:05.918380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:05.919014 (+ 634us) server_negotiation.cc:407] Connection header received
0806 04:52:05.919218 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:05.919224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:05.919299 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:05.919396 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:05.920320 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.921531 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.924868 (+ 3337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.926133 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.926265 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.926328 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:05.926810 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 04:52:05.926934 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:52:05.927446 21541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.917933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:46529 (local address 127.13.148.129:41927)
0806 04:52:05.918898 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:05.918916 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:05.919074 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:05.919422 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:05.919427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:05.919469 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:05.920115 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.920127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.921672 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.921676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.924676 (+ 3000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.924692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.926331 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.926339 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.926613 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.926682 (+ 69us) client_negotiation.cc:769] Sending connection context
0806 04:52:05.926774 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 04:52:05.926994 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":412,"spinlock_wait_cycles":30336,"thread_start_us":451,"threads_started":1}
I20260806 04:52:05.927968 21540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.917933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:46089 (local address 127.13.148.129:40931)
0806 04:52:05.918844 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:05.918888 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:05.919074 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:05.919422 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:05.919427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:05.919466 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:05.920114 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.920127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.921671 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.921676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.924676 (+ 3000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:05.924692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.926346 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.926350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:05.926608 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.926685 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 04:52:05.926774 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 04:52:05.926994 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":363,"spinlock_wait_cycles":48640,"thread_start_us":657,"threads_started":1}
I20260806 04:52:05.928505 21450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02416b6a1e5e4feb97e851ceda26aacd" candidate_uuid: "75e962bb879145bd8a56e7cc23142fdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b39b7889a4cd4aa781f52d643dec6432" is_pre_election: true
I20260806 04:52:05.928989 21307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02416b6a1e5e4feb97e851ceda26aacd" candidate_uuid: "75e962bb879145bd8a56e7cc23142fdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" is_pre_election: true
I20260806 04:52:05.929246 21450 raft_consensus.cc:2454] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75e962bb879145bd8a56e7cc23142fdf in term 0.
I20260806 04:52:05.929639 21307 raft_consensus.cc:2454] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75e962bb879145bd8a56e7cc23142fdf in term 0.
I20260806 04:52:05.930039 21089 leader_election.cc:302] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75e962bb879145bd8a56e7cc23142fdf, b39b7889a4cd4aa781f52d643dec6432; no voters:
I20260806 04:52:05.930474 21539 raft_consensus.cc:2781] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:52:05.930678 21539 raft_consensus.cc:482] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:52:05.930821 21539 raft_consensus.cc:3037] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:05.933116 21539 raft_consensus.cc:504] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } } peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } }
I20260806 04:52:05.933722 21539 leader_election.cc:288] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [CANDIDATE]: Term 1 election: Requested vote from peers b39b7889a4cd4aa781f52d643dec6432 (127.13.148.131:46529), 06e4f08b79e043fcae6f7a6ad7b8f281 (127.13.148.130:46089)
I20260806 04:52:05.934093 21450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02416b6a1e5e4feb97e851ceda26aacd" candidate_uuid: "75e962bb879145bd8a56e7cc23142fdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b39b7889a4cd4aa781f52d643dec6432"
I20260806 04:52:05.934182 21307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02416b6a1e5e4feb97e851ceda26aacd" candidate_uuid: "75e962bb879145bd8a56e7cc23142fdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281"
I20260806 04:52:05.934274 21450 raft_consensus.cc:3037] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:05.934391 21307 raft_consensus.cc:3037] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:05.936400 21307 raft_consensus.cc:2454] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75e962bb879145bd8a56e7cc23142fdf in term 1.
I20260806 04:52:05.936403 21450 raft_consensus.cc:2454] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75e962bb879145bd8a56e7cc23142fdf in term 1.
I20260806 04:52:05.936825 21092 leader_election.cc:302] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e4f08b79e043fcae6f7a6ad7b8f281, 75e962bb879145bd8a56e7cc23142fdf; no voters:
I20260806 04:52:05.937124 21539 raft_consensus.cc:2781] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:52:05.937553 21539 raft_consensus.cc:686] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [term 1 LEADER]: Becoming Leader. State: Replica: 75e962bb879145bd8a56e7cc23142fdf, State: Running, Role: LEADER
I20260806 04:52:05.938171 21539 consensus_queue.cc:237] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } } peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } }
I20260806 04:52:05.943264 20209 catalog_manager.cc:5495] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf reported cstate change: term changed from 0 to 1, leader changed from <none> to 75e962bb879145bd8a56e7cc23142fdf (127.13.148.129). New cstate: current_term: 1 leader_uuid: "75e962bb879145bd8a56e7cc23142fdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } health_report { overall_health: HEALTHY } } }
I20260806 04:52:05.974453 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:05.969493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.13.148.129:33753)
0806 04:52:05.969708 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:05.969716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:05.969785 (+ 69us) server_negotiation.cc:407] Connection header received
0806 04:52:05.969951 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:05.969954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:05.970034 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:05.970147 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:52:05.970473 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.971151 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.972432 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:05.972899 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:05.972990 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:05.973391 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:52:05.973891 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:52:05.973997 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:05.974094 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 04:52:05.974153 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 04:52:05.978052 21207 tablet.cc:2369] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:52:05.983167 21307 raft_consensus.cc:1261] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 1 FOLLOWER]: Refusing update from remote peer 75e962bb879145bd8a56e7cc23142fdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:52:05.983227 21450 raft_consensus.cc:1261] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432 [term 1 FOLLOWER]: Refusing update from remote peer 75e962bb879145bd8a56e7cc23142fdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:52:05.984122 21542 consensus_queue.cc:1035] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:52:05.984469 21539 consensus_queue.cc:1035] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:52:06.000924 21545 mvcc.cc:205] Tried to move back new op lower bound from 7315422928816320512 to 7315422928646545408. Current Snapshot: MvccSnapshot[applied={T|T < 7315422928816320512}]
I20260806 04:52:06.022867 20208 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:52:06.023209 20208 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:52:06.033169 20208 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8545f2e2779a47e3ac5cb4398a291b1b: Sending DeleteTablet for 3 replicas of tablet 02416b6a1e5e4feb97e851ceda26aacd
I20260806 04:52:06.034219 21430 tablet_service.cc:1514] Processing DeleteTablet for tablet 02416b6a1e5e4feb97e851ceda26aacd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:06 UTC) from {username='oryx', principal='oryx/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:58492
I20260806 04:52:06.035007 21287 tablet_service.cc:1514] Processing DeleteTablet for tablet 02416b6a1e5e4feb97e851ceda26aacd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:06 UTC) from {username='oryx', principal='oryx/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:48834
I20260806 04:52:06.035128 21137 tablet_service.cc:1514] Processing DeleteTablet for tablet 02416b6a1e5e4feb97e851ceda26aacd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:06 UTC) from {username='oryx', principal='oryx/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:33282
I20260806 04:52:06.055142 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 20178
I20260806 04:52:06.056190 20240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:06.062544 21558 tablet_replica.cc:330] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf: stopping tablet replica
I20260806 04:52:06.063118 21558 raft_consensus.cc:2229] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:52:06.063985 21558 raft_consensus.cc:2258] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:06.066155 21557 tablet_replica.cc:330] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281: stopping tablet replica
I20260806 04:52:06.066524 21558 ts_tablet_manager.cc:1905] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:06.066852 21557 raft_consensus.cc:2229] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:52:06.067492 21556 tablet_replica.cc:330] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432: stopping tablet replica
I20260806 04:52:06.068008 21557 raft_consensus.cc:2258] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:06.068488 21556 raft_consensus.cc:2229] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:52:06.069257 21556 raft_consensus.cc:2258] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:06.070989 21557 ts_tablet_manager.cc:1905] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:06.074676 21556 ts_tablet_manager.cc:1905] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:06.074735 21558 ts_tablet_manager.cc:1918] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:52:06.074914 21558 log.cc:1196] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/02416b6a1e5e4feb97e851ceda26aacd
I20260806 04:52:06.075546 21558 ts_tablet_manager.cc:1939] T 02416b6a1e5e4feb97e851ceda26aacd P 75e962bb879145bd8a56e7cc23142fdf: Deleting consensus metadata
I20260806 04:52:06.078274 21557 ts_tablet_manager.cc:1918] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:52:06.078456 21557 log.cc:1196] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/02416b6a1e5e4feb97e851ceda26aacd
I20260806 04:52:06.078928 21557 ts_tablet_manager.cc:1939] T 02416b6a1e5e4feb97e851ceda26aacd P 06e4f08b79e043fcae6f7a6ad7b8f281: Deleting consensus metadata
I20260806 04:52:06.087035 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:06.042047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.13.148.189:40159)
0806 04:52:06.042362 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:06.042369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:06.045881 (+ 3512us) server_negotiation.cc:407] Connection header received
0806 04:52:06.046159 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:06.046165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:06.046281 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:06.046351 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:06.062288 (+ 15937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:06.063431 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:06.067321 (+ 3890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:06.067955 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:06.068059 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:06.080718 (+ 12659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:06.080724 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:06.080729 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:06.080803 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:06.083804 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:06.084490 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:06.084513 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:06.084520 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:06.084662 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:06.085311 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:06.085316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:06.085332 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:06.085697 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:06.085925 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:06.086491 (+ 566us) server_negotiation.cc:299] Negotiation successful
0806 04:52:06.086719 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 04:52:06.091050 21556 ts_tablet_manager.cc:1918] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:52:06.091231 21556 log.cc:1196] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/02416b6a1e5e4feb97e851ceda26aacd
I20260806 04:52:06.091812 21556 ts_tablet_manager.cc:1939] T 02416b6a1e5e4feb97e851ceda26aacd P b39b7889a4cd4aa781f52d643dec6432: Deleting consensus metadata
I20260806 04:52:06.105460 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:06.048685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54878 (local address 127.13.148.188:39045)
0806 04:52:06.048916 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:06.048923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:06.088560 (+ 39637us) server_negotiation.cc:407] Connection header received
0806 04:52:06.088818 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:06.088825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:06.088922 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:06.089005 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:06.090782 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:06.091887 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:06.093741 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:06.094337 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:06.094474 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:06.097799 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:06.097807 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:06.097825 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:06.097868 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:06.102055 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:06.102821 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:06.102826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:06.102828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:06.103038 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:06.103518 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:06.103524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:06.103529 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:06.103968 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:06.104187 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:06.104731 (+ 544us) server_negotiation.cc:299] Negotiation successful
0806 04:52:06.105051 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:52:06.213747 20194 catalog_manager.cc:4841] TS b39b7889a4cd4aa781f52d643dec6432 (127.13.148.131:46529): tablet 02416b6a1e5e4feb97e851ceda26aacd (table test-table [id=c3b34ddfe02c487fbf65c0c6eb0cd78a]) successfully deleted
I20260806 04:52:06.217062 20197 catalog_manager.cc:4841] TS 06e4f08b79e043fcae6f7a6ad7b8f281 (127.13.148.130:46089): tablet 02416b6a1e5e4feb97e851ceda26aacd (table test-table [id=c3b34ddfe02c487fbf65c0c6eb0cd78a]) successfully deleted
I20260806 04:52:06.217396 20196 catalog_manager.cc:4841] TS 75e962bb879145bd8a56e7cc23142fdf (127.13.148.129:33753): tablet 02416b6a1e5e4feb97e851ceda26aacd (table test-table [id=c3b34ddfe02c487fbf65c0c6eb0cd78a]) successfully deleted
W20260806 04:52:06.217574 20196 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.13.148.129:33753, user_credentials={real_user=slave}} blocked reactor thread for 140170us
I20260806 04:52:06.218187 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 20178
I20260806 04:52:06.274794 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:06.250294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.13.148.188:39045)
0806 04:52:06.250501 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:06.250508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:06.260280 (+ 9772us) server_negotiation.cc:407] Connection header received
0806 04:52:06.260513 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:06.260522 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:06.260626 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:06.260732 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:06.262670 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:06.263383 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:06.265201 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:06.265647 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:06.265768 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:06.269620 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:06.269625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:06.269628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:06.269656 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:06.272269 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:06.272872 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:06.272877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:06.272879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:06.272992 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:06.273336 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:06.273340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:06.273343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:06.273641 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:06.273873 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:06.274298 (+ 425us) server_negotiation.cc:299] Negotiation successful
0806 04:52:06.274480 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 04:52:06.275910 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:06.250388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.13.148.189:40159)
0806 04:52:06.250573 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:06.250582 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:06.261235 (+ 10653us) server_negotiation.cc:407] Connection header received
0806 04:52:06.261379 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:06.261387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:06.261484 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:06.261603 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:06.263425 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:06.264115 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:06.265384 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:06.265958 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:06.266078 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:06.269588 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:06.269594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:06.269598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:06.269632 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:06.273013 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:06.273806 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:06.273810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:06.273814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:06.273960 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:06.274385 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:06.274405 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:06.274410 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:06.274727 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:06.274933 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:06.275427 (+ 494us) server_negotiation.cc:299] Negotiation successful
0806 04:52:06.275621 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 04:52:07.010159 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:07.009310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.130:43369)
0806 04:52:07.009795 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:07.009939 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":261,"threads_started":1}
W20260806 04:52:07.010735 21353 heartbeater.cc:644] Failed to heartbeat to 127.13.148.190:38833 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
I20260806 04:52:07.013247 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:07.012487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.131:35873)
0806 04:52:07.012977 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:07.013096 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":230,"threads_started":1}
W20260806 04:52:07.013578 21497 heartbeater.cc:644] Failed to heartbeat to 127.13.148.190:38833 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
I20260806 04:52:07.014621 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:07.014063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.129:57999)
0806 04:52:07.014385 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:07.014496 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":168,"threads_started":1}
W20260806 04:52:07.014935 21205 heartbeater.cc:644] Failed to heartbeat to 127.13.148.190:38833 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
I20260806 04:52:07.531286 21573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8545f2e2779a47e3ac5cb4398a291b1b)
I20260806 04:52:07.531278 21572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8545f2e2779a47e3ac5cb4398a291b1b)
I20260806 04:52:07.531569 21573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } }
I20260806 04:52:07.531584 21572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } }
I20260806 04:52:07.535732 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:07.533018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:50898)
0806 04:52:07.535188 (+ 2170us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:07.535394 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2039,"thread_start_us":1864,"threads_started":1}
I20260806 04:52:07.538920 21573 leader_election.cc:288] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8545f2e2779a47e3ac5cb4398a291b1b (127.13.148.190:38833), 14c97b2918954c2c90029c4234e84213 (127.13.148.188:39045)
I20260806 04:52:07.539958 21572 leader_election.cc:288] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8545f2e2779a47e3ac5cb4398a291b1b (127.13.148.190:38833), 324249b33cc748309596dc8e2ec5f26e (127.13.148.189:40159)
I20260806 04:52:07.544200 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:07.533015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:50906)
0806 04:52:07.543686 (+ 10671us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:07.543874 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10515,"threads_started":1}
W20260806 04:52:07.545634 20811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
I20260806 04:52:07.549425 21574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:07.548851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:50922)
0806 04:52:07.549062 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:07.549223 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260806 04:52:07.549641 20811 leader_election.cc:334] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8545f2e2779a47e3ac5cb4398a291b1b (127.13.148.190:38833): Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
W20260806 04:52:07.550503 21005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
I20260806 04:52:07.552608 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:07.552054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:50934)
0806 04:52:07.552224 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:07.552375 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 04:52:07.552842 21005 leader_election.cc:334] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8545f2e2779a47e3ac5cb4398a291b1b (127.13.148.190:38833): Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
I20260806 04:52:07.553894 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:07.533174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:40159 (local address 127.0.0.1:47850)
0806 04:52:07.544405 (+ 11231us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:07.544425 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:07.544571 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:07.546956 (+ 2385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:07.546963 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:07.547005 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:07.547345 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:07.547361 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:07.549247 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:07.549252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:07.551745 (+ 2493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:07.551761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:07.553254 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:07.553258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:07.553428 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:07.553455 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 04:52:07.553532 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 04:52:07.553620 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11127,"mutex_wait_us":30,"thread_start_us":11474,"threads_started":1}
I20260806 04:52:07.554059 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:07.535107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.13.148.189:40159)
0806 04:52:07.545656 (+ 10549us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:07.545662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:07.545679 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:52:07.545753 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:07.545760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:07.545821 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:07.545907 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:07.547620 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:07.549080 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:07.551910 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:07.553107 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:07.553246 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:07.553314 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:07.553576 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 04:52:07.553700 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":10445,"thread_start_us":10268,"threads_started":1}
I20260806 04:52:07.554538 20838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14c97b2918954c2c90029c4234e84213" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "324249b33cc748309596dc8e2ec5f26e" is_pre_election: true
I20260806 04:52:07.554834 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:07.544922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54898 (local address 127.13.148.188:39045)
0806 04:52:07.547213 (+ 2291us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:07.547222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:07.547249 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:52:07.547326 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:07.547334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:07.547441 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:07.548275 (+ 834us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:07.548587 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:07.550169 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:07.552507 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:07.553683 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:07.553858 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:07.553931 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:07.554422 (+ 491us) server_negotiation.cc:299] Negotiation successful
0806 04:52:07.554540 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":2113,"thread_start_us":3463,"threads_started":1}
I20260806 04:52:07.554883 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:07.533044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:39045 (local address 127.0.0.1:54898)
0806 04:52:07.537292 (+ 4248us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:07.537316 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:07.537467 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:07.547631 (+ 10164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:07.547639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:07.547676 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:07.548099 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:07.548115 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:07.550386 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:07.550392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:07.552332 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:07.552351 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:07.553869 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:07.553877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:07.554261 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:07.554293 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 04:52:07.554401 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 04:52:07.554539 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4093,"mutex_wait_us":20,"thread_start_us":13481,"threads_started":1}
I20260806 04:52:07.554857 20838 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c97b2918954c2c90029c4234e84213 in term 1.
I20260806 04:52:07.555498 21007 leader_election.cc:302] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14c97b2918954c2c90029c4234e84213, 324249b33cc748309596dc8e2ec5f26e; no voters: 8545f2e2779a47e3ac5cb4398a291b1b
I20260806 04:52:07.555533 21033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "324249b33cc748309596dc8e2ec5f26e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "14c97b2918954c2c90029c4234e84213" is_pre_election: true
I20260806 04:52:07.555835 21033 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 324249b33cc748309596dc8e2ec5f26e in term 1.
I20260806 04:52:07.556011 21572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:52:07.556147 21572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8545f2e2779a47e3ac5cb4398a291b1b)
I20260806 04:52:07.556257 21572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:52:07.556298 20810 leader_election.cc:302] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14c97b2918954c2c90029c4234e84213, 324249b33cc748309596dc8e2ec5f26e; no voters: 8545f2e2779a47e3ac5cb4398a291b1b
I20260806 04:52:07.556700 21573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:52:07.556851 21573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8545f2e2779a47e3ac5cb4398a291b1b)
I20260806 04:52:07.556951 21573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:52:07.559307 21572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } }
I20260806 04:52:07.559801 21573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } }
I20260806 04:52:07.560012 21572 leader_election.cc:288] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [CANDIDATE]: Term 2 election: Requested vote from peers 8545f2e2779a47e3ac5cb4398a291b1b (127.13.148.190:38833), 324249b33cc748309596dc8e2ec5f26e (127.13.148.189:40159)
I20260806 04:52:07.560474 21573 leader_election.cc:288] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [CANDIDATE]: Term 2 election: Requested vote from peers 8545f2e2779a47e3ac5cb4398a291b1b (127.13.148.190:38833), 14c97b2918954c2c90029c4234e84213 (127.13.148.188:39045)
I20260806 04:52:07.560524 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:07.560147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:50948)
0806 04:52:07.560263 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:07.560359 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260806 04:52:07.560523 20838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14c97b2918954c2c90029c4234e84213" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "324249b33cc748309596dc8e2ec5f26e"
I20260806 04:52:07.560840 20838 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 14c97b2918954c2c90029c4234e84213 in current term 2: Already voted for candidate 324249b33cc748309596dc8e2ec5f26e in this term.
I20260806 04:52:07.560926 21033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "324249b33cc748309596dc8e2ec5f26e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "14c97b2918954c2c90029c4234e84213"
I20260806 04:52:07.561193 21033 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 324249b33cc748309596dc8e2ec5f26e in current term 2: Already voted for candidate 14c97b2918954c2c90029c4234e84213 in this term.
I20260806 04:52:07.561909 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:07.561551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:50960)
0806 04:52:07.561698 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:07.561796 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 04:52:07.562098 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:07.561630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:50956)
0806 04:52:07.561878 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:07.561935 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
W20260806 04:52:07.562124 21005 leader_election.cc:334] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8545f2e2779a47e3ac5cb4398a291b1b (127.13.148.190:38833): Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
I20260806 04:52:07.562217 21005 leader_election.cc:302] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14c97b2918954c2c90029c4234e84213; no voters: 324249b33cc748309596dc8e2ec5f26e, 8545f2e2779a47e3ac5cb4398a291b1b
I20260806 04:52:07.562463 21572 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:52:07.562964 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:07.562725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:50972)
0806 04:52:07.562822 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:07.562876 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 04:52:07.563141 20811 leader_election.cc:334] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8545f2e2779a47e3ac5cb4398a291b1b (127.13.148.190:38833): Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
I20260806 04:52:07.563223 20811 leader_election.cc:302] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 324249b33cc748309596dc8e2ec5f26e; no voters: 14c97b2918954c2c90029c4234e84213, 8545f2e2779a47e3ac5cb4398a291b1b
I20260806 04:52:07.563410 21573 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:52:08.012414 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:08.011412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.130:50363)
0806 04:52:08.012064 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:08.012175 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":416,"threads_started":1}
I20260806 04:52:08.014756 21583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:08.014188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.131:46381)
0806 04:52:08.014541 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:08.014609 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":184,"threads_started":1}
I20260806 04:52:08.016009 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:08.015407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.129:44439)
0806 04:52:08.015798 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:08.015868 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":228,"threads_started":1}
I20260806 04:52:09.015058 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.013438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.130:44845)
0806 04:52:09.014726 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.014829 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1176,"thread_start_us":1296,"threads_started":1}
I20260806 04:52:09.016471 21586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.015510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.131:39829)
0806 04:52:09.016193 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.016263 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":528,"threads_started":1}
I20260806 04:52:09.017362 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.016726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.129:57793)
0806 04:52:09.017139 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.017213 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":236,"threads_started":1}
I20260806 04:52:09.451259 21588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:52:09.451452 21588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } }
I20260806 04:52:09.452137 21588 leader_election.cc:288] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8545f2e2779a47e3ac5cb4398a291b1b (127.13.148.190:38833), 324249b33cc748309596dc8e2ec5f26e (127.13.148.189:40159)
I20260806 04:52:09.452664 20838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14c97b2918954c2c90029c4234e84213" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "324249b33cc748309596dc8e2ec5f26e" is_pre_election: true
I20260806 04:52:09.452955 20838 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c97b2918954c2c90029c4234e84213 in term 2.
I20260806 04:52:09.453037 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.452192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:35182)
0806 04:52:09.452658 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.452787 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":218,"threads_started":1}
I20260806 04:52:09.453374 21007 leader_election.cc:302] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c97b2918954c2c90029c4234e84213, 324249b33cc748309596dc8e2ec5f26e; no voters:
I20260806 04:52:09.453639 21588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:52:09.453774 21588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:52:09.453889 21588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:52:09.454311 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.454023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:35196)
0806 04:52:09.454141 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.454209 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260806 04:52:09.454478 21005 leader_election.cc:334] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8545f2e2779a47e3ac5cb4398a291b1b (127.13.148.190:38833): Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
I20260806 04:52:09.455897 21588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } }
I20260806 04:52:09.456408 21588 leader_election.cc:288] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [CANDIDATE]: Term 3 election: Requested vote from peers 8545f2e2779a47e3ac5cb4398a291b1b (127.13.148.190:38833), 324249b33cc748309596dc8e2ec5f26e (127.13.148.189:40159)
I20260806 04:52:09.456939 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.456478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:35202)
0806 04:52:09.456635 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.456751 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 04:52:09.456892 20838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14c97b2918954c2c90029c4234e84213" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "324249b33cc748309596dc8e2ec5f26e"
I20260806 04:52:09.457069 20838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:52:09.457815 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.457525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:35208)
0806 04:52:09.457656 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.457723 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 04:52:09.457952 21005 leader_election.cc:334] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8545f2e2779a47e3ac5cb4398a291b1b (127.13.148.190:38833): Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
I20260806 04:52:09.459280 20838 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c97b2918954c2c90029c4234e84213 in term 3.
I20260806 04:52:09.459621 21007 leader_election.cc:302] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14c97b2918954c2c90029c4234e84213, 324249b33cc748309596dc8e2ec5f26e; no voters: 8545f2e2779a47e3ac5cb4398a291b1b
I20260806 04:52:09.459885 21588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:52:09.460261 21588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [term 3 LEADER]: Becoming Leader. State: Replica: 14c97b2918954c2c90029c4234e84213, State: Running, Role: LEADER
I20260806 04:52:09.460623 21588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } }
I20260806 04:52:09.462860 21591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14c97b2918954c2c90029c4234e84213. Latest consensus state: current_term: 3 leader_uuid: "14c97b2918954c2c90029c4234e84213" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } } }
I20260806 04:52:09.463073 21591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:09.463629 21593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:52:09.466924 21593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:52:09.467676 21593 catalog_manager.cc:1259] Loaded cluster ID: b888a0e01c47497bb126f3d75f810621
I20260806 04:52:09.467762 21593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:52:09.468828 21593 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:52:09.469597 21593 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: Loaded TSK: 0
I20260806 04:52:09.470366 21593 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:52:09.534170 21022 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:52:09.541126 21022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:52:09.545182 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.544765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:35218)
0806 04:52:09.544998 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.545062 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260806 04:52:09.545228 20838 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [term 3 FOLLOWER]: Refusing update from remote peer 14c97b2918954c2c90029c4234e84213: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260806 04:52:09.546134 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.545809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:35222)
0806 04:52:09.545968 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.546034 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 04:52:09.546001 21591 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260806 04:52:09.546365 21005 consensus_peers.cc:487] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 -> Peer 8545f2e2779a47e3ac5cb4398a291b1b (127.13.148.190:38833): Couldn't send request to peer 8545f2e2779a47e3ac5cb4398a291b1b. Status: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 04:52:09.548437 21595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14c97b2918954c2c90029c4234e84213. Latest consensus state: current_term: 3 leader_uuid: "14c97b2918954c2c90029c4234e84213" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } } }
I20260806 04:52:09.548679 21595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:52:09.550657 21595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "14c97b2918954c2c90029c4234e84213" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } } }
I20260806 04:52:09.550947 21595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324249b33cc748309596dc8e2ec5f26e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:52:09.552570 21600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "14c97b2918954c2c90029c4234e84213" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8545f2e2779a47e3ac5cb4398a291b1b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38833 } } peers { permanent_uuid: "324249b33cc748309596dc8e2ec5f26e" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 40159 } attrs { promote: false } } peers { permanent_uuid: "14c97b2918954c2c90029c4234e84213" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 39045 } attrs { promote: false } } }
I20260806 04:52:09.552999 21600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213 [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:09.572863 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.558511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:46089 (local address 127.0.0.1:52000)
0806 04:52:09.558706 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.558723 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:09.558841 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:09.566943 (+ 8102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:09.566947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:09.566969 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:09.567232 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:09.567243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.568564 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.568570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:09.570487 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:09.570505 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.572069 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.572075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:09.572241 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:09.572276 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 04:52:09.572368 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 04:52:09.572488 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 04:52:09.573122 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.558603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.13.148.130:46089)
0806 04:52:09.566693 (+ 8090us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:09.566697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:09.566710 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:52:09.566761 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:09.566765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:09.566811 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:09.566907 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:09.567376 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.568400 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.570716 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.571893 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.572049 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:09.572117 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:09.572407 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 04:52:09.572536 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8006,"thread_start_us":7945,"threads_started":1}
I20260806 04:52:09.573474 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.559301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:46529 (local address 127.0.0.1:57320)
0806 04:52:09.566314 (+ 7013us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.566328 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:09.566414 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:09.567592 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:09.567596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:09.567615 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:09.567951 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:09.567962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.569188 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.569193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:09.571318 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:09.571333 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.572615 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.572620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:09.572814 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:09.572862 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 04:52:09.572927 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 04:52:09.573049 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6921,"mutex_wait_us":109,"thread_start_us":6779,"threads_started":1}
I20260806 04:52:09.573730 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.558912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:33753 (local address 127.0.0.1:45658)
0806 04:52:09.559466 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.559485 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:09.559609 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:09.568271 (+ 8662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:09.568275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:09.568294 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:09.568566 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:09.568574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.569756 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.569765 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:09.571354 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:09.571370 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.572869 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.572875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:09.573028 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:09.573057 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 04:52:09.573127 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 04:52:09.573295 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":49,"thread_start_us":534,"threads_started":1}
I20260806 04:52:09.574092 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.559320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.13.148.129:33753)
0806 04:52:09.568026 (+ 8706us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:09.568033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:09.568049 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:52:09.568108 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:09.568113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:09.568173 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:09.568261 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:09.568713 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.569624 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.571554 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.572614 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.573597 (+ 983us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:09.573676 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:09.573730 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 04:52:09.573858 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8590,"thread_start_us":8497,"threads_started":1}
I20260806 04:52:09.574198 21287 tablet_service.cc:1467] Processing CreateTablet for tablet 2fdb4c5655c84ba5b200d0d67d2e40c9 (DEFAULT_TABLE table=test-table [id=563d62091502406aa2837c49bbaf8f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:09.574416 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.559397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57320 (local address 127.13.148.131:46529)
0806 04:52:09.567350 (+ 7953us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:09.567355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:09.567369 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:52:09.567437 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:09.567440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:09.567489 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:09.567580 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:09.568102 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.569039 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.571481 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.572443 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.573873 (+ 1430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:09.573942 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:09.573991 (+ 49us) server_negotiation.cc:299] Negotiation successful
0806 04:52:09.574126 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7830,"thread_start_us":7745,"threads_started":1}
I20260806 04:52:09.574641 21287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fdb4c5655c84ba5b200d0d67d2e40c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:09.575019 21430 tablet_service.cc:1467] Processing CreateTablet for tablet 2fdb4c5655c84ba5b200d0d67d2e40c9 (DEFAULT_TABLE table=test-table [id=563d62091502406aa2837c49bbaf8f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:09.575366 21430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fdb4c5655c84ba5b200d0d67d2e40c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:09.578037 21137 tablet_service.cc:1467] Processing CreateTablet for tablet 2fdb4c5655c84ba5b200d0d67d2e40c9 (DEFAULT_TABLE table=test-table [id=563d62091502406aa2837c49bbaf8f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:09.578426 21137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fdb4c5655c84ba5b200d0d67d2e40c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:09.586258 21607 tablet_bootstrap.cc:492] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432: Bootstrap starting.
I20260806 04:52:09.588452 21607 tablet_bootstrap.cc:654] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:09.589182 21608 tablet_bootstrap.cc:492] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf: Bootstrap starting.
I20260806 04:52:09.589289 21606 tablet_bootstrap.cc:492] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281: Bootstrap starting.
I20260806 04:52:09.590685 21608 tablet_bootstrap.cc:654] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:09.590809 21607 tablet_bootstrap.cc:492] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432: No bootstrap required, opened a new log
I20260806 04:52:09.590921 21606 tablet_bootstrap.cc:654] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:09.590979 21607 ts_tablet_manager.cc:1397] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 04:52:09.591832 21607 raft_consensus.cc:348] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } } peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } }
I20260806 04:52:09.592095 21607 raft_consensus.cc:374] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:09.592211 21607 raft_consensus.cc:729] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b39b7889a4cd4aa781f52d643dec6432, State: Initialized, Role: FOLLOWER
I20260806 04:52:09.592461 21607 consensus_queue.cc:260] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } } peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } }
I20260806 04:52:09.592590 21608 tablet_bootstrap.cc:492] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf: No bootstrap required, opened a new log
I20260806 04:52:09.592728 21608 ts_tablet_manager.cc:1397] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260806 04:52:09.593170 21607 ts_tablet_manager.cc:1428] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 04:52:09.593487 21608 raft_consensus.cc:348] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } } peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } }
I20260806 04:52:09.593739 21608 raft_consensus.cc:374] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:09.593832 21608 raft_consensus.cc:729] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e962bb879145bd8a56e7cc23142fdf, State: Initialized, Role: FOLLOWER
I20260806 04:52:09.594072 21608 consensus_queue.cc:260] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } } peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } }
I20260806 04:52:09.595320 21608 ts_tablet_manager.cc:1428] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 04:52:09.598762 21496 heartbeater.cc:503] Master 127.13.148.188:39045 was elected leader, sending a full tablet report...
I20260806 04:52:09.599620 21204 heartbeater.cc:503] Master 127.13.148.188:39045 was elected leader, sending a full tablet report...
I20260806 04:52:09.604501 21606 tablet_bootstrap.cc:492] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281: No bootstrap required, opened a new log
I20260806 04:52:09.604780 21606 ts_tablet_manager.cc:1397] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260806 04:52:09.605636 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.593401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.131:50063)
0806 04:52:09.605286 (+ 11885us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.605421 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11730,"thread_start_us":11625,"threads_started":1}
I20260806 04:52:09.605855 21606 raft_consensus.cc:348] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } } peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } }
I20260806 04:52:09.606208 21606 raft_consensus.cc:374] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:09.606345 21606 raft_consensus.cc:729] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e4f08b79e043fcae6f7a6ad7b8f281, State: Initialized, Role: FOLLOWER
I20260806 04:52:09.606730 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.606285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.131:41977)
0806 04:52:09.606528 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.606612 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260806 04:52:09.606695 21606 consensus_queue.cc:260] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } } peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } }
I20260806 04:52:09.607650 21606 ts_tablet_manager.cc:1428] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:52:09.609265 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.596068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.129:53471)
0806 04:52:09.608902 (+ 12834us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.609046 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12690,"threads_started":1}
I20260806 04:52:09.610008 21352 heartbeater.cc:503] Master 127.13.148.188:39045 was elected leader, sending a full tablet report...
I20260806 04:52:09.610447 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.610083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.129:48933)
0806 04:52:09.610279 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.610335 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260806 04:52:09.613796 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.607724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.130:52815)
0806 04:52:09.613381 (+ 5657us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.613549 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5549,"thread_start_us":5389,"threads_started":1}
I20260806 04:52:09.614887 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.614229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.130:47007)
0806 04:52:09.614594 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.614724 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":186,"threads_started":1}
I20260806 04:52:09.825513 21616 raft_consensus.cc:482] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:52:09.825762 21616 raft_consensus.cc:504] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } } peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } }
I20260806 04:52:09.827312 21616 leader_election.cc:288] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75e962bb879145bd8a56e7cc23142fdf (127.13.148.129:33753), b39b7889a4cd4aa781f52d643dec6432 (127.13.148.131:46529)
I20260806 04:52:09.833194 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.827526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:33753 (local address 127.13.148.130:34555)
0806 04:52:09.827781 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.827800 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:09.827948 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:09.828287 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:09.828290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:09.828309 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:09.828609 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:09.828622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.829785 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.829792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:09.831352 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:09.831364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.832436 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.832443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:09.832620 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:09.832656 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 04:52:09.832814 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 04:52:09.832912 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":79}
I20260806 04:52:09.833230 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.827749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:34555 (local address 127.13.148.129:33753)
0806 04:52:09.827931 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:09.827935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:09.827950 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:52:09.828086 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:09.828092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:09.828175 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:09.828267 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:09.828820 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.829673 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.831505 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.832266 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.832375 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:09.832420 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:09.832847 (+ 427us) server_negotiation.cc:299] Negotiation successful
0806 04:52:09.832977 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 04:52:09.833433 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.827653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:48723 (local address 127.13.148.131:46529)
0806 04:52:09.827843 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:09.827849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:09.828167 (+ 318us) server_negotiation.cc:407] Connection header received
0806 04:52:09.828395 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:09.828402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:09.828474 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:09.828574 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:09.829064 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.829716 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.831584 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.832516 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.832643 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:09.832715 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:09.833075 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 04:52:09.833194 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:52:09.833417 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.827572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:46529 (local address 127.13.148.130:48723)
0806 04:52:09.828070 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.828086 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:09.828212 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:09.828597 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:09.828600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:09.828635 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:09.828945 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:09.828956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.829842 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.829845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:09.831388 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:09.831400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.832713 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.832717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:09.832916 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:09.832946 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 04:52:09.833021 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 04:52:09.833140 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":66}
I20260806 04:52:09.834254 21450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fdb4c5655c84ba5b200d0d67d2e40c9" candidate_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b39b7889a4cd4aa781f52d643dec6432" is_pre_election: true
I20260806 04:52:09.834457 21450 raft_consensus.cc:2454] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06e4f08b79e043fcae6f7a6ad7b8f281 in term 0.
I20260806 04:52:09.834359 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fdb4c5655c84ba5b200d0d67d2e40c9" candidate_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75e962bb879145bd8a56e7cc23142fdf" is_pre_election: true
I20260806 04:52:09.834774 21158 raft_consensus.cc:2454] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06e4f08b79e043fcae6f7a6ad7b8f281 in term 0.
I20260806 04:52:09.834885 21239 leader_election.cc:302] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e4f08b79e043fcae6f7a6ad7b8f281, b39b7889a4cd4aa781f52d643dec6432; no voters:
I20260806 04:52:09.835211 21616 raft_consensus.cc:2781] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:52:09.835325 21616 raft_consensus.cc:482] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:52:09.835419 21616 raft_consensus.cc:3037] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:09.837670 21616 raft_consensus.cc:504] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } } peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } }
I20260806 04:52:09.838208 21616 leader_election.cc:288] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [CANDIDATE]: Term 1 election: Requested vote from peers 75e962bb879145bd8a56e7cc23142fdf (127.13.148.129:33753), b39b7889a4cd4aa781f52d643dec6432 (127.13.148.131:46529)
I20260806 04:52:09.838579 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fdb4c5655c84ba5b200d0d67d2e40c9" candidate_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75e962bb879145bd8a56e7cc23142fdf"
I20260806 04:52:09.838755 21158 raft_consensus.cc:3037] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:09.838703 21450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fdb4c5655c84ba5b200d0d67d2e40c9" candidate_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b39b7889a4cd4aa781f52d643dec6432"
I20260806 04:52:09.838902 21450 raft_consensus.cc:3037] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:09.841061 21158 raft_consensus.cc:2454] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06e4f08b79e043fcae6f7a6ad7b8f281 in term 1.
I20260806 04:52:09.841073 21450 raft_consensus.cc:2454] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06e4f08b79e043fcae6f7a6ad7b8f281 in term 1.
I20260806 04:52:09.841425 21241 leader_election.cc:302] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e4f08b79e043fcae6f7a6ad7b8f281, 75e962bb879145bd8a56e7cc23142fdf; no voters:
I20260806 04:52:09.841697 21616 raft_consensus.cc:2781] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:52:09.842136 21616 raft_consensus.cc:686] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 1 LEADER]: Becoming Leader. State: Replica: 06e4f08b79e043fcae6f7a6ad7b8f281, State: Running, Role: LEADER
I20260806 04:52:09.842737 21616 consensus_queue.cc:237] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } } peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } }
I20260806 04:52:09.843101 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.842676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.130:39917)
0806 04:52:09.842814 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.842931 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 04:52:09.846197 21022 catalog_manager.cc:5495] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06e4f08b79e043fcae6f7a6ad7b8f281 (127.13.148.130). New cstate: current_term: 1 leader_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e4f08b79e043fcae6f7a6ad7b8f281" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 } health_report { overall_health: UNKNOWN } } }
I20260806 04:52:09.927583 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.923040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.13.148.130:46089)
0806 04:52:09.923210 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:09.923214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:09.923231 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:52:09.923357 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:09.923361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:09.923429 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:09.923543 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:52:09.923853 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.924567 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.925759 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.926149 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.926236 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:09.926533 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:52:09.927022 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:52:09.927118 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:09.927228 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 04:52:09.927306 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:52:09.931682 21450 raft_consensus.cc:1261] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432 [term 1 FOLLOWER]: Refusing update from remote peer 06e4f08b79e043fcae6f7a6ad7b8f281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:52:09.932037 21158 raft_consensus.cc:1261] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf [term 1 FOLLOWER]: Refusing update from remote peer 06e4f08b79e043fcae6f7a6ad7b8f281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:52:09.932375 21619 consensus_queue.cc:1035] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b39b7889a4cd4aa781f52d643dec6432" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 46529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:52:09.932739 21616 consensus_queue.cc:1035] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75e962bb879145bd8a56e7cc23142fdf" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 33753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:52:09.934850 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.934287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.131:51907)
0806 04:52:09.934511 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.934625 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260806 04:52:09.935056 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.934554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.129:36779)
0806 04:52:09.934798 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.934879 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260806 04:52:09.943802 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.943398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.131:50065)
0806 04:52:09.943579 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.943673 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 04:52:09.948303 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.944721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.129:39375)
0806 04:52:09.948087 (+ 3366us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.948175 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3241}
I20260806 04:52:09.952839 21622 mvcc.cc:205] Tried to move back new op lower bound from 7315422944992645120 to 7315422944639463424. Current Snapshot: MvccSnapshot[applied={T|T < 7315422944992645120 or (T in {7315422944992645120})}]
I20260806 04:52:09.953210 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.952520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.130:46959)
0806 04:52:09.952849 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.953001 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260806 04:52:09.956478 21022 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:52:09.957010 21022 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:52:09.957199 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.956316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.13.148.130:37981)
0806 04:52:09.956959 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:09.957036 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525}
I20260806 04:52:09.962555 21022 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14c97b2918954c2c90029c4234e84213: Sending DeleteTablet for 3 replicas of tablet 2fdb4c5655c84ba5b200d0d67d2e40c9
I20260806 04:52:09.963477 21287 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fdb4c5655c84ba5b200d0d67d2e40c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:09 UTC) from {username='oryx', principal='oryx/127.13.148.188@KRBTEST.COM'} at 127.0.0.1:52000
I20260806 04:52:09.964083 21632 tablet_replica.cc:330] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281: stopping tablet replica
I20260806 04:52:09.964339 21632 raft_consensus.cc:2229] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:52:09.964509 21137 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fdb4c5655c84ba5b200d0d67d2e40c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:09 UTC) from {username='oryx', principal='oryx/127.13.148.188@KRBTEST.COM'} at 127.0.0.1:45658
I20260806 04:52:09.964922 21632 raft_consensus.cc:2258] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:09.965212 21633 tablet_replica.cc:330] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf: stopping tablet replica
I20260806 04:52:09.965425 21633 raft_consensus.cc:2229] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:52:09.965766 21633 raft_consensus.cc:2258] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:09.966034 21632 ts_tablet_manager.cc:1905] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:09.966852 21430 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fdb4c5655c84ba5b200d0d67d2e40c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:09 UTC) from {username='oryx', principal='oryx/127.13.148.188@KRBTEST.COM'} at 127.0.0.1:57320
I20260806 04:52:09.967056 21633 ts_tablet_manager.cc:1905] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:09.970558 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 20997
I20260806 04:52:09.972744 21634 tablet_replica.cc:330] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432: stopping tablet replica
I20260806 04:52:09.972754 21632 ts_tablet_manager.cc:1918] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:52:09.972925 21632 log.cc:1196] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/2fdb4c5655c84ba5b200d0d67d2e40c9
I20260806 04:52:09.973083 21634 raft_consensus.cc:2229] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:52:09.973424 21632 ts_tablet_manager.cc:1939] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 06e4f08b79e043fcae6f7a6ad7b8f281: Deleting consensus metadata
I20260806 04:52:09.974092 21633 ts_tablet_manager.cc:1918] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:52:09.974395 21634 raft_consensus.cc:2258] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:09.974453 21007 catalog_manager.cc:4841] TS 06e4f08b79e043fcae6f7a6ad7b8f281 (127.13.148.130:46089): tablet 2fdb4c5655c84ba5b200d0d67d2e40c9 (table test-table [id=563d62091502406aa2837c49bbaf8f17]) successfully deleted
I20260806 04:52:09.974895 21633 log.cc:1196] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/2fdb4c5655c84ba5b200d0d67d2e40c9
I20260806 04:52:09.975394 21634 ts_tablet_manager.cc:1905] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:09.979478 21634 ts_tablet_manager.cc:1918] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:52:09.979604 21634 log.cc:1196] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/2fdb4c5655c84ba5b200d0d67d2e40c9
I20260806 04:52:09.979996 21633 ts_tablet_manager.cc:1939] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P 75e962bb879145bd8a56e7cc23142fdf: Deleting consensus metadata
I20260806 04:52:09.980033 21634 ts_tablet_manager.cc:1939] T 2fdb4c5655c84ba5b200d0d67d2e40c9 P b39b7889a4cd4aa781f52d643dec6432: Deleting consensus metadata
I20260806 04:52:09.981158 21004 catalog_manager.cc:4841] TS b39b7889a4cd4aa781f52d643dec6432 (127.13.148.131:46529): tablet 2fdb4c5655c84ba5b200d0d67d2e40c9 (table test-table [id=563d62091502406aa2837c49bbaf8f17]) successfully deleted
I20260806 04:52:09.981185 21006 catalog_manager.cc:4841] TS 75e962bb879145bd8a56e7cc23142fdf (127.13.148.129:33753): tablet 2fdb4c5655c84ba5b200d0d67d2e40c9 (table test-table [id=563d62091502406aa2837c49bbaf8f17]) successfully deleted
I20260806 04:52:09.992890 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:09.972204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.13.148.129:33753)
0806 04:52:09.972375 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:09.972381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:09.981496 (+ 9115us) server_negotiation.cc:407] Connection header received
0806 04:52:09.981731 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:09.981737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:09.981818 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:09.981916 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:09.983035 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.983710 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.984819 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:09.985181 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:09.985304 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:09.987618 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:09.987629 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:09.987643 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:09.987672 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:09.990284 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:09.990917 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:09.990923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:09.990925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:09.991062 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:09.991416 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:09.991423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:09.991428 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:09.991768 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:09.992072 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:09.992325 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 04:52:09.992521 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 04:52:09.993441 21199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:10.038519 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:10.037957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:35238)
0806 04:52:10.038214 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:10.038333 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260806 04:52:10.039825 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:10.039401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:35250)
0806 04:52:10.039586 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:10.039674 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 04:52:10.123127 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 20997
I20260806 04:52:10.155077 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 21216
I20260806 04:52:10.169497 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:10.155796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.13.148.130:46089)
0806 04:52:10.155980 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:10.155986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:10.156013 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:52:10.156178 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:10.156184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:10.156270 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:10.156378 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:10.157783 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:10.158904 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:10.160010 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:10.160579 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:10.160718 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:10.163195 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:10.163208 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:10.163226 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:10.163268 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:10.166878 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:10.167421 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:10.167429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:10.167433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:10.167597 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:10.167985 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:10.167989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:10.167993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:10.168454 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:10.168758 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:10.168934 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 04:52:10.169152 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:52:10.170174 21347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:10.282799 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 21216
I20260806 04:52:10.303854 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 21365
I20260806 04:52:10.317525 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:10.304555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.13.148.131:46529)
0806 04:52:10.304729 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:10.304737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:10.304819 (+ 82us) server_negotiation.cc:407] Connection header received
0806 04:52:10.305055 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:10.305060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:10.305125 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:10.305222 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:10.306589 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:10.307252 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:10.308426 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:10.308946 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:10.309074 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:10.311667 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:10.311684 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:10.311701 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:10.311762 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:10.314629 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:10.315327 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:10.315333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:10.315359 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:10.315493 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:10.315870 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:10.315878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:10.315882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:10.316236 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:10.316568 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:10.316914 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 04:52:10.317152 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:52:10.318274 21491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:10.409070 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 21365
I20260806 04:52:10.428676 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 20683
I20260806 04:52:10.429854 20859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:10.521194 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:10.520637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:35256)
0806 04:52:10.520874 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:10.521004 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 04:52:10.522279 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:10.521929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:38833 (local address 127.0.0.1:35270)
0806 04:52:10.522072 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:10.522149 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 04:52:10.543285 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 20683
I20260806 04:52:10.565366 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 20797
I20260806 04:52:10.566326 21053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:10.680544 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 20797
2026-08-06T04:52:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11291 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:52:10 dist-test-slave-1x4g krb5kdc[21643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:52:10 dist-test-slave-1x4g krb5kdc[21643](info): set up 2 sockets
Aug 06 04:52:10 dist-test-slave-1x4g krb5kdc[21643](info): commencing operation
krb5kdc: starting...
W20260806 04:52:11.718307 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:52:11 dist-test-slave-1x4g krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:52:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:52:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.190.keytab.
Entry for principal oryx/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.190.keytab.
I20260806 04:52:11.919723 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:45859
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.190.keytab
--principal=oryx/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:45859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:12.220579 21656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:12.220902 21656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:12.221076 21656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:12.227360 21656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:52:12.227460 21656 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 04:52:12.227527 21656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:12.227643 21656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:12.227728 21656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:52:12.227844 21656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:52:12.237262 21656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:45859
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.190.keytab
--principal=oryx/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:45859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:12.238602 21656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:12.239971 21656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:12.251883 21661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:12.251820 21662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:12.251856 21664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:12.252678 21656 server_base.cc:1034] running on GCE node
I20260806 04:52:12.253415 21656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:12.254657 21656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:12.255863 21656 hybrid_clock.cc:648] HybridClock initialized: now 1785991932255808 us; error 35 us; skew 500 ppm
I20260806 04:52:12.256134 21656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:12 dist-test-slave-1x4g krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991932, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:12.259090 21656 init.cc:377] Logged in from keytab as oryx/127.13.148.190@KRBTEST.COM (short username oryx)
I20260806 04:52:12.260406 21656 webserver.cc:460] Webserver started at http://127.13.148.190:33743/ using document root <none> and password file <none>
I20260806 04:52:12.261145 21656 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:12.261256 21656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:12.261592 21656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:12.264326 21656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "63d61c9de3024b839c4d051e0679a7a6"
format_stamp: "Formatted at 2026-08-06 04:52:12 on dist-test-slave-1x4g"
server_key: "0a75750ac4615cd7f9e39756ac06e45d"
server_key_iv: "1341ff3ceb08570411c90fa0cd9b6703"
server_key_version: "encryptionkey@0"
I20260806 04:52:12.265203 21656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "63d61c9de3024b839c4d051e0679a7a6"
format_stamp: "Formatted at 2026-08-06 04:52:12 on dist-test-slave-1x4g"
server_key: "0a75750ac4615cd7f9e39756ac06e45d"
server_key_iv: "1341ff3ceb08570411c90fa0cd9b6703"
server_key_version: "encryptionkey@0"
I20260806 04:52:12.270150 21656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 04:52:12.273514 21671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:12.274772 21656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:52:12.274953 21656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "63d61c9de3024b839c4d051e0679a7a6"
format_stamp: "Formatted at 2026-08-06 04:52:12 on dist-test-slave-1x4g"
server_key: "0a75750ac4615cd7f9e39756ac06e45d"
server_key_iv: "1341ff3ceb08570411c90fa0cd9b6703"
server_key_version: "encryptionkey@0"
I20260806 04:52:12.275175 21656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:12.291800 21656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:12.295149 21656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:12.295375 21656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:12.316159 21656 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:45859
I20260806 04:52:12.316190 21733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:45859 every 8 connection(s)
I20260806 04:52:12.317466 21656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:52:12.321108 21735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:12.326184 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 21656
I20260806 04:52:12.326385 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:52:12.327039 13906 external_mini_cluster.cc:1428] Setting key 205f5f20ee4b76fdd3c9bd7c862cce77
I20260806 04:52:12.331903 21735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6: Bootstrap starting.
I20260806 04:52:12.335326 21735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:12.336438 21735 log.cc:824] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6: Log is configured to *not* fsync() on all Append() calls
I20260806 04:52:12.339334 21735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6: No bootstrap required, opened a new log
Aug 06 04:52:12 dist-test-slave-1x4g krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM
I20260806 04:52:12.348248 21735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d61c9de3024b839c4d051e0679a7a6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45859 } }
I20260806 04:52:12.348627 21735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:12.348721 21735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d61c9de3024b839c4d051e0679a7a6, State: Initialized, Role: FOLLOWER
I20260806 04:52:12.349298 21735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d61c9de3024b839c4d051e0679a7a6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45859 } }
I20260806 04:52:12.349537 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:12.329026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.13.148.190:45859)
0806 04:52:12.329806 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:12.329819 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:12.329862 (+ 43us) server_negotiation.cc:407] Connection header received
0806 04:52:12.332067 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:12.332089 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:12.332529 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:12.332924 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:12.334036 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:12.335931 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:12.337399 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:12.338255 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:12.338413 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:12.342095 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:12.342105 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:12.342122 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:12.342171 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:12.345719 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:12.346288 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:12.346296 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:12.346303 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:12.346528 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:12.346838 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:12.346842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:12.346845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:12.347484 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:12.347736 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:12.348020 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 04:52:12.348364 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":333,"threads_started":1}
I20260806 04:52:12.349586 21735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:52:12.349694 21735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:52:12.349862 21735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:12.351974 21735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d61c9de3024b839c4d051e0679a7a6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45859 } }
I20260806 04:52:12.352463 21735 leader_election.cc:302] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63d61c9de3024b839c4d051e0679a7a6; no voters:
I20260806 04:52:12.353011 21735 leader_election.cc:288] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:52:12.353152 21740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:52:12.353854 21740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [term 1 LEADER]: Becoming Leader. State: Replica: 63d61c9de3024b839c4d051e0679a7a6, State: Running, Role: LEADER
I20260806 04:52:12.354343 21740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d61c9de3024b839c4d051e0679a7a6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45859 } }
I20260806 04:52:12.354858 21735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:52:12.356904 21742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63d61c9de3024b839c4d051e0679a7a6. Latest consensus state: current_term: 1 leader_uuid: "63d61c9de3024b839c4d051e0679a7a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d61c9de3024b839c4d051e0679a7a6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45859 } } }
I20260806 04:52:12.357364 21742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:12.358484 21741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63d61c9de3024b839c4d051e0679a7a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d61c9de3024b839c4d051e0679a7a6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 45859 } } }
I20260806 04:52:12.358880 21741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6 [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:12.359206 21748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:52:12.363739 21748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:52:12.370801 21748 catalog_manager.cc:1347] Generated new cluster ID: effcad9702c045ab9a9d743a442f90b6
I20260806 04:52:12.370895 21748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:52:12.378883 21748 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:52:12.380278 21748 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:52:12.385982 21748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6: Generated new TSK 0
I20260806 04:52:12.386560 21748 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.129.keytab.
Entry for principal oryx/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.129.keytab.
I20260806 04:52:12.436223 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:45859
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.129.keytab
--principal=oryx/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:12.748646 21761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:12.748939 21761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:12.749105 21761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:12.755950 21761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:12.756068 21761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:12.756186 21761 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:52:12.756301 21761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:52:12.765223 21761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.129.keytab
--principal=oryx/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.190:45859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:12.766235 21761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:12.767362 21761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:12.779726 21766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:12.779778 21769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:12.779748 21767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:12.780536 21761 server_base.cc:1034] running on GCE node
I20260806 04:52:12.781150 21761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:12.782006 21761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:12.783206 21761 hybrid_clock.cc:648] HybridClock initialized: now 1785991932783144 us; error 44 us; skew 500 ppm
I20260806 04:52:12.783495 21761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:12 dist-test-slave-1x4g krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991932, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:12.786401 21761 init.cc:377] Logged in from keytab as oryx/127.13.148.129@KRBTEST.COM (short username oryx)
I20260806 04:52:12.787624 21761 webserver.cc:460] Webserver started at http://127.13.148.129:33267/ using document root <none> and password file <none>
I20260806 04:52:12.788275 21761 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:12.788395 21761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:12.788689 21761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:12.791198 21761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "476a7eb6447240c7a21612c339917d90"
format_stamp: "Formatted at 2026-08-06 04:52:12 on dist-test-slave-1x4g"
server_key: "6b4c8db71cf208f88f6f93b84b42456d"
server_key_iv: "367340607063ae12335e09db422decca"
server_key_version: "encryptionkey@0"
I20260806 04:52:12.792011 21761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "476a7eb6447240c7a21612c339917d90"
format_stamp: "Formatted at 2026-08-06 04:52:12 on dist-test-slave-1x4g"
server_key: "6b4c8db71cf208f88f6f93b84b42456d"
server_key_iv: "367340607063ae12335e09db422decca"
server_key_version: "encryptionkey@0"
I20260806 04:52:12.796622 21761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 04:52:12.799837 21776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:12.801036 21761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:52:12.801198 21761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "476a7eb6447240c7a21612c339917d90"
format_stamp: "Formatted at 2026-08-06 04:52:12 on dist-test-slave-1x4g"
server_key: "6b4c8db71cf208f88f6f93b84b42456d"
server_key_iv: "367340607063ae12335e09db422decca"
server_key_version: "encryptionkey@0"
I20260806 04:52:12.801409 21761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:12.812593 21761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:12.815971 21761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:12.816216 21761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:12.818622 21761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:12.818722 21761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:12.818881 21761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:12.818964 21761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:52:12 dist-test-slave-1x4g krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991932, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM
I20260806 04:52:12.842831 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:12.819390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:51601 (local address 127.13.148.190:45859)
0806 04:52:12.819566 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:12.819590 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:12.821105 (+ 1515us) server_negotiation.cc:407] Connection header received
0806 04:52:12.823184 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:12.823191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:12.823277 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:12.823415 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:12.825953 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:12.826896 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:12.829486 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:12.830193 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:12.830341 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:12.835132 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:12.835140 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:12.835146 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:12.835181 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:12.838975 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:12.840023 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:12.840032 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:12.840038 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:12.840220 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:12.840666 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:12.840672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:12.840678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:12.841119 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:12.841305 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:12.842280 (+ 975us) server_negotiation.cc:299] Negotiation successful
0806 04:52:12.842486 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:52:12.843868 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:12.819948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:45859 (local address 127.13.148.129:51601)
0806 04:52:12.820941 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:12.820988 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:12.822918 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:12.823834 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:12.823841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:12.824690 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:12.825669 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:12.825717 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:12.827099 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:12.827105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:12.829347 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:12.829363 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:12.830666 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:12.830673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:12.831033 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:12.831787 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:12.831818 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:12.834836 (+ 3018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:12.839154 (+ 4318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:12.839165 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:12.839182 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:12.839860 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:12.840364 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:12.840367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:12.840369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:12.840548 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:12.841282 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:12.841292 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:12.841736 (+ 444us) client_negotiation.cc:769] Sending connection context
0806 04:52:12.842187 (+ 451us) client_negotiation.cc:240] Negotiation successful
0806 04:52:12.842686 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":458,"threads_started":1}
I20260806 04:52:12.865039 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:12.848983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:55303 (local address 127.13.148.190:45859)
0806 04:52:12.849110 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:12.849113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:12.849705 (+ 592us) server_negotiation.cc:407] Connection header received
0806 04:52:12.849893 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:12.849903 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:12.849988 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:12.850084 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:12.851521 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:12.852908 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:12.855423 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:12.856004 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:12.856108 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:12.858501 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:12.858506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:12.858510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:12.858544 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:12.861959 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:12.862951 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:12.862957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:12.862962 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:12.863126 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:12.863647 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:12.863652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:12.863655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:12.863910 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:12.864061 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:12.864463 (+ 402us) server_negotiation.cc:299] Negotiation successful
0806 04:52:12.864732 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 04:52:12.865118 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:12.849063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:45859 (local address 127.13.148.129:55303)
0806 04:52:12.849614 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:12.849627 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:12.849762 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:12.850135 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:12.850144 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:12.850771 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:12.851299 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:12.851318 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:12.853136 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:12.853144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:12.855261 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:12.855283 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:12.856181 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:12.856188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:12.856433 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:12.857287 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:12.857316 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:12.858319 (+ 1003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:12.862139 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:12.862149 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:12.862153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:12.862785 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:12.863266 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:12.863275 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:12.863278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:12.863487 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:12.864025 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:12.864029 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:12.864201 (+ 172us) client_negotiation.cc:769] Sending connection context
0806 04:52:12.864358 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 04:52:12.864658 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":305,"threads_started":1}
I20260806 04:52:12.872031 21761 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:40165
I20260806 04:52:12.872052 21897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:40165 every 8 connection(s)
I20260806 04:52:12.873232 21761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:52:12.875131 21898 heartbeater.cc:347] Connected to a master server at 127.13.148.190:45859
I20260806 04:52:12.875178 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 21761
I20260806 04:52:12.875347 21898 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:12.875411 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:52:12.875929 13906 external_mini_cluster.cc:1428] Setting key 4166a79d36d822d2a545b99261686f47
I20260806 04:52:12.875941 21898 heartbeater.cc:511] Master 127.13.148.190:45859 requested a full tablet report, sending...
I20260806 04:52:12.877885 21688 ts_manager.cc:194] Registered new tserver with Master: 476a7eb6447240c7a21612c339917d90 (127.13.148.129:40165)
I20260806 04:52:12.879509 21688 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:51601
WARNING: no policy specified for oryx/127.13.148.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.130.keytab.
Entry for principal oryx/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.130.keytab.
I20260806 04:52:12.921831 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:45859
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.130.keytab
--principal=oryx/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:13.218191 21902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:13.218431 21902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:13.218577 21902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:13.224947 21902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:13.225054 21902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:13.225193 21902 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:52:13.225306 21902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:52:13.233474 21902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.130.keytab
--principal=oryx/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.190:45859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:13.234553 21902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:13.235775 21902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:13.247814 21908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:13.247834 21911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:13.247772 21909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:13.248507 21902 server_base.cc:1034] running on GCE node
I20260806 04:52:13.249082 21902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:13.249989 21902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:13.251200 21902 hybrid_clock.cc:648] HybridClock initialized: now 1785991933251132 us; error 33 us; skew 500 ppm
I20260806 04:52:13.251484 21902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:13 dist-test-slave-1x4g krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991933, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:13.254587 21902 init.cc:377] Logged in from keytab as oryx/127.13.148.130@KRBTEST.COM (short username oryx)
I20260806 04:52:13.255908 21902 webserver.cc:460] Webserver started at http://127.13.148.130:34641/ using document root <none> and password file <none>
I20260806 04:52:13.256552 21902 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:13.256706 21902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:13.257043 21902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:13.259860 21902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "46f4d9c56fa54125b72c2256a2e86937"
format_stamp: "Formatted at 2026-08-06 04:52:13 on dist-test-slave-1x4g"
server_key: "fdcc1645e31e1433aecc73b6373f4a9b"
server_key_iv: "e47a8249d3608bd7e9d9261a5b30cd4c"
server_key_version: "encryptionkey@0"
I20260806 04:52:13.260592 21902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "46f4d9c56fa54125b72c2256a2e86937"
format_stamp: "Formatted at 2026-08-06 04:52:13 on dist-test-slave-1x4g"
server_key: "fdcc1645e31e1433aecc73b6373f4a9b"
server_key_iv: "e47a8249d3608bd7e9d9261a5b30cd4c"
server_key_version: "encryptionkey@0"
I20260806 04:52:13.265355 21902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 04:52:13.268323 21918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:13.269673 21902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:52:13.269824 21902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "46f4d9c56fa54125b72c2256a2e86937"
format_stamp: "Formatted at 2026-08-06 04:52:13 on dist-test-slave-1x4g"
server_key: "fdcc1645e31e1433aecc73b6373f4a9b"
server_key_iv: "e47a8249d3608bd7e9d9261a5b30cd4c"
server_key_version: "encryptionkey@0"
I20260806 04:52:13.270025 21902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:13.295353 21902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:13.299501 21902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:13.299746 21902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:13.302211 21902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:13.302336 21902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:13.302474 21902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:13.302543 21902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:52:13 dist-test-slave-1x4g krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991933, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM
I20260806 04:52:13.325637 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:13.303041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:45859 (local address 127.13.148.130:52207)
0806 04:52:13.303877 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:13.303927 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:13.306178 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:13.306885 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:13.306897 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:13.307546 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:13.308351 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:13.308378 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.309706 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.309711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:13.310939 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:13.310965 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.311988 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.311999 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:13.312340 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:13.313060 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:13.313079 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:13.316195 (+ 3116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:13.320085 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:13.320093 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:13.320106 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:13.320585 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:13.321122 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:13.321133 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:13.321139 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:13.321425 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:13.322528 (+ 1103us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:13.322538 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:13.323020 (+ 482us) client_negotiation.cc:769] Sending connection context
0806 04:52:13.323363 (+ 343us) client_negotiation.cc:240] Negotiation successful
0806 04:52:13.323962 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":361,"threads_started":1}
I20260806 04:52:13.326591 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:13.302573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:52207 (local address 127.13.148.190:45859)
0806 04:52:13.302742 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:13.302748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:13.304055 (+ 1307us) server_negotiation.cc:407] Connection header received
0806 04:52:13.306438 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:13.306446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:13.306547 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:13.306661 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:13.308543 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.309544 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.311096 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.311694 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.311841 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:13.316477 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:13.316483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:13.316487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:13.316526 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:13.319881 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:13.320720 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:13.320727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:13.320733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:13.320957 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:13.321785 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:13.321793 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:13.321799 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:13.322269 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:13.322423 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:13.325304 (+ 2881us) server_negotiation.cc:299] Negotiation successful
0806 04:52:13.325522 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 04:52:13.350644 21982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:13.336176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:45859 (local address 127.13.148.130:34069)
0806 04:52:13.336868 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:13.336884 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:13.337034 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:13.337419 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:13.337430 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:13.338110 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:13.338506 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:13.338520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.339535 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.339543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:13.341624 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:13.341644 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.342597 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.342607 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:13.342843 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:13.343561 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:13.343589 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:13.344577 (+ 988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:13.347990 (+ 3413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:13.347994 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:13.347998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:13.348321 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:13.348655 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:13.348659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:13.348662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:13.348815 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:13.349317 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:13.349322 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:13.349547 (+ 225us) client_negotiation.cc:769] Sending connection context
0806 04:52:13.349707 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 04:52:13.350012 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":367,"threads_started":1}
I20260806 04:52:13.351414 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:13.336042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:34069 (local address 127.13.148.190:45859)
0806 04:52:13.336208 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:13.336213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:13.336970 (+ 757us) server_negotiation.cc:407] Connection header received
0806 04:52:13.337168 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:13.337175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:13.337264 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:13.337365 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:13.338718 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.339402 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.341852 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.342369 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.342498 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:13.344803 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:13.344810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:13.344814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:13.344849 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:13.347875 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:13.348424 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:13.348429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:13.348432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:13.348549 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:13.348901 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:13.348907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:13.348910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:13.349198 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:13.349340 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:13.350804 (+ 1464us) server_negotiation.cc:299] Negotiation successful
0806 04:52:13.351107 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:52:13.358809 21902 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:42645
I20260806 04:52:13.358844 22039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:42645 every 8 connection(s)
I20260806 04:52:13.359917 21902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:52:13.360901 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 21902
I20260806 04:52:13.361126 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:52:13.361712 13906 external_mini_cluster.cc:1428] Setting key d7e63c6fc9343e1984e6599c1d1560b1
I20260806 04:52:13.362082 22040 heartbeater.cc:347] Connected to a master server at 127.13.148.190:45859
I20260806 04:52:13.362300 22040 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:13.362903 22040 heartbeater.cc:511] Master 127.13.148.190:45859 requested a full tablet report, sending...
I20260806 04:52:13.364053 21688 ts_manager.cc:194] Registered new tserver with Master: 46f4d9c56fa54125b72c2256a2e86937 (127.13.148.130:42645)
I20260806 04:52:13.364848 21688 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:52207
WARNING: no policy specified for oryx/127.13.148.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.131.keytab.
Entry for principal oryx/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.131.keytab.
I20260806 04:52:13.406129 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:45859
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.131.keytab
--principal=oryx/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:13.711525 22044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:13.711776 22044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:13.711941 22044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:13.718586 22044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:13.718686 22044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:13.718843 22044 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:52:13.718979 22044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:52:13.727698 22044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.131.keytab
--principal=oryx/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.190:45859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:13.728995 22044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:13.730319 22044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:13.742436 22049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:13.742487 22050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:13.742892 22052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:13.743301 22044 server_base.cc:1034] running on GCE node
I20260806 04:52:13.743865 22044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:13.744644 22044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:13.745875 22044 hybrid_clock.cc:648] HybridClock initialized: now 1785991933745796 us; error 40 us; skew 500 ppm
I20260806 04:52:13.746203 22044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:13 dist-test-slave-1x4g krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991933, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:13.749239 22044 init.cc:377] Logged in from keytab as oryx/127.13.148.131@KRBTEST.COM (short username oryx)
I20260806 04:52:13.750489 22044 webserver.cc:460] Webserver started at http://127.13.148.131:35033/ using document root <none> and password file <none>
I20260806 04:52:13.751128 22044 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:13.751214 22044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:13.751469 22044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:13.754246 22044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "53d1b1fef3514a09bb73d4a956107a36"
format_stamp: "Formatted at 2026-08-06 04:52:13 on dist-test-slave-1x4g"
server_key: "467dcf5ca7313b485f2d86fffc8fefbd"
server_key_iv: "15c5dd2ce8e20a6ac0ced7c35ac52d51"
server_key_version: "encryptionkey@0"
I20260806 04:52:13.754914 22044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "53d1b1fef3514a09bb73d4a956107a36"
format_stamp: "Formatted at 2026-08-06 04:52:13 on dist-test-slave-1x4g"
server_key: "467dcf5ca7313b485f2d86fffc8fefbd"
server_key_iv: "15c5dd2ce8e20a6ac0ced7c35ac52d51"
server_key_version: "encryptionkey@0"
I20260806 04:52:13.759505 22044 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:52:13.762905 22059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:13.764107 22044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:52:13.764220 22044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "53d1b1fef3514a09bb73d4a956107a36"
format_stamp: "Formatted at 2026-08-06 04:52:13 on dist-test-slave-1x4g"
server_key: "467dcf5ca7313b485f2d86fffc8fefbd"
server_key_iv: "15c5dd2ce8e20a6ac0ced7c35ac52d51"
server_key_version: "encryptionkey@0"
I20260806 04:52:13.764369 22044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:13.788004 22044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:13.791864 22044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:13.792130 22044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:13.794512 22044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:13.794590 22044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:13.794755 22044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:13.794818 22044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:52:13 dist-test-slave-1x4g krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991933, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM
I20260806 04:52:13.818404 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:13.795346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:52879 (local address 127.13.148.190:45859)
0806 04:52:13.795553 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:13.795559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:13.796910 (+ 1351us) server_negotiation.cc:407] Connection header received
0806 04:52:13.798912 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:13.798922 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:13.799024 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:13.799173 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:13.801268 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.802336 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.804751 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.805383 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.805540 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:13.810598 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:13.810603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:13.810606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:13.810635 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:13.813964 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:13.815104 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:13.815111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:13.815114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:13.815288 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:13.815915 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:13.815922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:13.815928 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:13.816308 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:13.816476 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:13.817689 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0806 04:52:13.817974 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 04:52:13.820027 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:13.795810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:45859 (local address 127.13.148.131:52879)
0806 04:52:13.796719 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:13.796796 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:13.798639 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:13.799509 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:13.799525 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:13.800068 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:13.801073 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:13.801105 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.802496 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.802501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:13.804572 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:13.804583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.805645 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.805652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:13.805956 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:13.806751 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:13.806792 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:13.810308 (+ 3516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:13.814121 (+ 3813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:13.814131 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:13.814150 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:13.814935 (+ 785us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:13.815435 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:13.815454 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:13.815460 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:13.815759 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:13.816552 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:13.816565 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:13.817146 (+ 581us) client_negotiation.cc:769] Sending connection context
0806 04:52:13.817550 (+ 404us) client_negotiation.cc:240] Negotiation successful
0806 04:52:13.818183 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":361,"threads_started":1}
I20260806 04:52:13.835842 22044 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:43631
I20260806 04:52:13.835857 22180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:43631 every 8 connection(s)
I20260806 04:52:13.837455 22044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:52:13.839461 22181 heartbeater.cc:347] Connected to a master server at 127.13.148.190:45859
I20260806 04:52:13.839715 22181 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:13.840461 22181 heartbeater.cc:511] Master 127.13.148.190:45859 requested a full tablet report, sending...
I20260806 04:52:13.841055 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:13.825998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:45859 (local address 127.13.148.131:58027)
0806 04:52:13.826705 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:13.826726 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:13.826958 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:13.827320 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:13.827325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:13.827902 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:13.828236 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:13.828246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.829694 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.829701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:13.831759 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:13.831779 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.832687 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.832694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:13.832958 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:13.833712 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:13.833728 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:13.834593 (+ 865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:13.838384 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:13.838391 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:13.838397 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:13.839000 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:13.839511 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:13.839514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:13.839517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:13.839692 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:13.840193 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:13.840196 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:13.840334 (+ 138us) client_negotiation.cc:769] Sending connection context
0806 04:52:13.840440 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 04:52:13.840671 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":405,"threads_started":1}
I20260806 04:52:13.841192 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:13.825817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:58027 (local address 127.13.148.190:45859)
0806 04:52:13.825972 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:13.825976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:13.826831 (+ 855us) server_negotiation.cc:407] Connection header received
0806 04:52:13.827112 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:13.827117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:13.827197 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:13.827292 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:13.828404 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.829520 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.831939 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.832460 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.832620 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:13.834751 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:13.834756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:13.834761 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:13.834799 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:13.838182 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:13.839153 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:13.839158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:13.839162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:13.839387 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:13.839800 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:13.839805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:13.839807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:13.840095 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:13.840232 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:13.840590 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 04:52:13.840826 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:52:13.842144 21689 ts_manager.cc:194] Registered new tserver with Master: 53d1b1fef3514a09bb73d4a956107a36 (127.13.148.131:43631)
I20260806 04:52:13.843223 21689 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:52879
I20260806 04:52:13.844693 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 22044
I20260806 04:52:13.844902 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:52:13.845403 13906 external_mini_cluster.cc:1428] Setting key 6c57e5768d1b11627507acd5d6a5c597
I20260806 04:52:13.848009 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 04:52:13 dist-test-slave-1x4g krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260806 04:52:13.882124 21898 heartbeater.cc:503] Master 127.13.148.190:45859 was elected leader, sending a full tablet report...
Aug 06 04:52:13 dist-test-slave-1x4g krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:13 dist-test-slave-1x4g krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 04:52:13.896697 21738 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:13.887397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.13.148.190:45859)
0806 04:52:13.887695 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:13.887701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:13.887857 (+ 156us) server_negotiation.cc:407] Connection header received
0806 04:52:13.887982 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:13.887986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:13.888042 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:13.888116 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:13.889385 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.890046 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.891159 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.891580 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.891668 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:13.896461 (+ 4793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32962: BlockingRecv error: recv got EOF from 127.0.0.1:32962 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Aug 06 04:52:13 dist-test-slave-1x4g krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM
I20260806 04:52:13.910780 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:13.899618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32972 (local address 127.13.148.190:45859)
0806 04:52:13.899760 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:13.899763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:13.900078 (+ 315us) server_negotiation.cc:407] Connection header received
0806 04:52:13.900185 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:13.900188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:13.900236 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:13.900305 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:13.901464 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.902141 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.903292 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.903661 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.903737 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:13.906787 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:13.906791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:13.906794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:13.906820 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:13.908880 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:13.909368 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:13.909372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:13.909374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:13.909497 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:13.909789 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:13.909791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:13.909793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:13.910036 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:13.910215 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:13.910415 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 04:52:13.910573 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:52:13.914314 21689 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:52:13.920553 21689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:52:13.942651 22202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:13.932440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34644 (local address 127.13.148.129:40165)
0806 04:52:13.933013 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:13.933024 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:13.933875 (+ 851us) server_negotiation.cc:407] Connection header received
0806 04:52:13.934320 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:13.934331 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:13.934618 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:13.934796 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:13.935798 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.937958 (+ 2160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.940132 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.941569 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.941754 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:13.941932 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:13.942247 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 04:52:13.942395 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":269,"threads_started":1}
I20260806 04:52:13.942682 22201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:13.931969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:40165 (local address 127.0.0.1:34644)
0806 04:52:13.933706 (+ 1737us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:13.933756 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:13.934059 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:13.935144 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:13.935153 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:13.935187 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:13.935610 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:13.935624 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.938123 (+ 2499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.938130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:13.939980 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:13.939990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.941787 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.941795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:13.942003 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:13.942084 (+ 81us) client_negotiation.cc:769] Sending connection context
0806 04:52:13.942206 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 04:52:13.942368 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1391,"thread_start_us":4539,"threads_started":1}
I20260806 04:52:13.944319 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:13.932351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:43631 (local address 127.0.0.1:49942)
0806 04:52:13.936823 (+ 4472us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:13.936841 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:13.936949 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:13.937921 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:13.937927 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:13.937951 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:13.938441 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:13.938453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.940067 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.940074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:13.941902 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:13.941924 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.943575 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.943581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:13.943715 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:13.943751 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 04:52:13.943862 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 04:52:13.943966 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4331,"mutex_wait_us":3,"thread_start_us":843,"threads_started":1}
I20260806 04:52:13.945591 22205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:13.937867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:42645 (local address 127.0.0.1:32958)
0806 04:52:13.938453 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:13.938468 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:13.938607 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:13.939196 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:13.939200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:13.939222 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:13.939540 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:13.939549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.941151 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.941158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:13.943235 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:13.943249 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.944919 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.944924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:13.945027 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:13.945046 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:52:13.945168 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 04:52:13.945328 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":273,"threads_started":1}
I20260806 04:52:13.945786 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:13.937190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.13.148.130:42645)
0806 04:52:13.937848 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:13.937854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:13.938570 (+ 716us) server_negotiation.cc:407] Connection header received
0806 04:52:13.938818 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:13.938826 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:13.939011 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:13.939192 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:13.939698 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.940977 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.943387 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.944417 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.944563 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:13.944710 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:13.945356 (+ 646us) server_negotiation.cc:299] Negotiation successful
0806 04:52:13.945539 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":379,"threads_started":1}
I20260806 04:52:13.946856 21832 tablet_service.cc:1467] Processing CreateTablet for tablet 40c8f1a166e54ad6bbad94e9031ab38b (DEFAULT_TABLE table=test-table [id=76d0083dfcfc4a1e999a330b07d6e4dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:13.948019 21832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c8f1a166e54ad6bbad94e9031ab38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:13.948390 21972 tablet_service.cc:1467] Processing CreateTablet for tablet 40c8f1a166e54ad6bbad94e9031ab38b (DEFAULT_TABLE table=test-table [id=76d0083dfcfc4a1e999a330b07d6e4dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:13.949465 21972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c8f1a166e54ad6bbad94e9031ab38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:13.953338 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:13.932656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49942 (local address 127.13.148.131:43631)
0806 04:52:13.936347 (+ 3691us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:13.936358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:13.937083 (+ 725us) server_negotiation.cc:407] Connection header received
0806 04:52:13.937193 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:13.937199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:13.937435 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:13.937622 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:13.938678 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.939952 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.942089 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:13.943382 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:13.943574 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:13.943720 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:13.943902 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 04:52:13.953031 (+ 9129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3495,"thread_start_us":3271,"threads_started":1}
I20260806 04:52:13.956171 22110 tablet_service.cc:1467] Processing CreateTablet for tablet 40c8f1a166e54ad6bbad94e9031ab38b (DEFAULT_TABLE table=test-table [id=76d0083dfcfc4a1e999a330b07d6e4dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:13.957481 22110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c8f1a166e54ad6bbad94e9031ab38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:13.958411 22206 tablet_bootstrap.cc:492] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90: Bootstrap starting.
I20260806 04:52:13.961400 22206 tablet_bootstrap.cc:654] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:13.962154 22207 tablet_bootstrap.cc:492] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937: Bootstrap starting.
I20260806 04:52:13.962672 22206 log.cc:824] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90: Log is configured to *not* fsync() on all Append() calls
I20260806 04:52:13.965034 22207 tablet_bootstrap.cc:654] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:13.965469 22206 tablet_bootstrap.cc:492] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90: No bootstrap required, opened a new log
I20260806 04:52:13.965787 22206 ts_tablet_manager.cc:1397] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 04:52:13.966276 22207 log.cc:824] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937: Log is configured to *not* fsync() on all Append() calls
I20260806 04:52:13.968861 22207 tablet_bootstrap.cc:492] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937: No bootstrap required, opened a new log
I20260806 04:52:13.969163 22207 ts_tablet_manager.cc:1397] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260806 04:52:13.969434 22210 tablet_bootstrap.cc:492] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36: Bootstrap starting.
I20260806 04:52:13.971894 22210 tablet_bootstrap.cc:654] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:13.972862 22210 log.cc:824] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36: Log is configured to *not* fsync() on all Append() calls
I20260806 04:52:13.975175 22210 tablet_bootstrap.cc:492] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36: No bootstrap required, opened a new log
I20260806 04:52:13.975390 22210 ts_tablet_manager.cc:1397] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:52:13.975302 22206 raft_consensus.cc:348] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } }
I20260806 04:52:13.975677 22206 raft_consensus.cc:374] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:13.975764 22206 raft_consensus.cc:729] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476a7eb6447240c7a21612c339917d90, State: Initialized, Role: FOLLOWER
I20260806 04:52:13.976471 22206 consensus_queue.cc:260] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } }
I20260806 04:52:13.979014 22207 raft_consensus.cc:348] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } }
I20260806 04:52:13.979450 22207 raft_consensus.cc:374] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:13.979558 22207 raft_consensus.cc:729] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46f4d9c56fa54125b72c2256a2e86937, State: Initialized, Role: FOLLOWER
I20260806 04:52:13.980237 22207 consensus_queue.cc:260] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } }
I20260806 04:52:13.981405 22040 heartbeater.cc:503] Master 127.13.148.190:45859 was elected leader, sending a full tablet report...
I20260806 04:52:13.982272 22207 ts_tablet_manager.cc:1428] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260806 04:52:13.982968 22210 raft_consensus.cc:348] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } }
I20260806 04:52:13.983355 22210 raft_consensus.cc:374] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:13.983491 22210 raft_consensus.cc:729] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1b1fef3514a09bb73d4a956107a36, State: Initialized, Role: FOLLOWER
I20260806 04:52:13.983614 22206 ts_tablet_manager.cc:1428] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20260806 04:52:13.984083 22210 consensus_queue.cc:260] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } }
I20260806 04:52:13.985083 22181 heartbeater.cc:503] Master 127.13.148.190:45859 was elected leader, sending a full tablet report...
I20260806 04:52:13.985643 22210 ts_tablet_manager.cc:1428] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260806 04:52:14.080042 22212 raft_consensus.cc:482] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:52:14.080297 22212 raft_consensus.cc:504] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } }
I20260806 04:52:14.081871 22212 leader_election.cc:288] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53d1b1fef3514a09bb73d4a956107a36 (127.13.148.131:43631), 46f4d9c56fa54125b72c2256a2e86937 (127.13.148.130:42645)
W20260806 04:52:14.089120 22182 tablet.cc:2369] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:52:14.089787 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:14.082313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:46313 (local address 127.13.148.131:43631)
0806 04:52:14.082521 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:14.082527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:14.083364 (+ 837us) server_negotiation.cc:407] Connection header received
0806 04:52:14.083498 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:14.083510 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:14.083592 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:14.083678 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:14.084344 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.085364 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.087635 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.088824 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.088993 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.089067 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:14.089405 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 04:52:14.089551 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:52:14.089872 22215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:14.082412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:43631 (local address 127.13.148.129:46313)
0806 04:52:14.083258 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:14.083273 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:14.083387 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:14.083701 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:14.083705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:14.083727 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:14.084192 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:14.084202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.085486 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.085491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:14.087432 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:14.087445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.089027 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.089032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:14.089199 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.089237 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 04:52:14.089371 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 04:52:14.089501 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"spinlock_wait_cycles":21760,"thread_start_us":456,"threads_started":1}
I20260806 04:52:14.089936 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:14.082398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:48069 (local address 127.13.148.130:42645)
0806 04:52:14.082596 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:14.082603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:14.083515 (+ 912us) server_negotiation.cc:407] Connection header received
0806 04:52:14.083663 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:14.083669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:14.083740 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:14.083834 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:14.084480 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.085567 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.087751 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.088886 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.089029 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.089091 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:14.089599 (+ 508us) server_negotiation.cc:299] Negotiation successful
0806 04:52:14.089712 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 04:52:14.090543 22216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:14.082455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:42645 (local address 127.13.148.129:48069)
0806 04:52:14.083385 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:14.083402 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:14.083527 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:14.083865 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:14.083871 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:14.083898 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:14.084289 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:14.084303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.085712 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.085717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:14.087608 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:14.087624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.089249 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.089257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:14.089410 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.089448 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 04:52:14.090094 (+ 646us) client_negotiation.cc:240] Negotiation successful
0806 04:52:14.090210 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":5,"spinlock_wait_cycles":896,"thread_start_us":426,"threads_started":1}
I20260806 04:52:14.090844 22131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c8f1a166e54ad6bbad94e9031ab38b" candidate_uuid: "476a7eb6447240c7a21612c339917d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53d1b1fef3514a09bb73d4a956107a36" is_pre_election: true
I20260806 04:52:14.091343 22131 raft_consensus.cc:2454] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 476a7eb6447240c7a21612c339917d90 in term 0.
I20260806 04:52:14.091463 21993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c8f1a166e54ad6bbad94e9031ab38b" candidate_uuid: "476a7eb6447240c7a21612c339917d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46f4d9c56fa54125b72c2256a2e86937" is_pre_election: true
I20260806 04:52:14.091977 21993 raft_consensus.cc:2454] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 476a7eb6447240c7a21612c339917d90 in term 0.
I20260806 04:52:14.092017 21779 leader_election.cc:302] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 476a7eb6447240c7a21612c339917d90, 53d1b1fef3514a09bb73d4a956107a36; no voters:
I20260806 04:52:14.092399 22212 raft_consensus.cc:2781] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:52:14.092520 22212 raft_consensus.cc:482] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:52:14.092625 22212 raft_consensus.cc:3037] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:14.094782 22212 raft_consensus.cc:504] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } }
I20260806 04:52:14.095434 22212 leader_election.cc:288] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [CANDIDATE]: Term 1 election: Requested vote from peers 53d1b1fef3514a09bb73d4a956107a36 (127.13.148.131:43631), 46f4d9c56fa54125b72c2256a2e86937 (127.13.148.130:42645)
I20260806 04:52:14.095753 22131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c8f1a166e54ad6bbad94e9031ab38b" candidate_uuid: "476a7eb6447240c7a21612c339917d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53d1b1fef3514a09bb73d4a956107a36"
I20260806 04:52:14.095933 22131 raft_consensus.cc:3037] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:14.095875 21993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c8f1a166e54ad6bbad94e9031ab38b" candidate_uuid: "476a7eb6447240c7a21612c339917d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46f4d9c56fa54125b72c2256a2e86937"
I20260806 04:52:14.096094 21993 raft_consensus.cc:3037] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:14.098287 22131 raft_consensus.cc:2454] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 476a7eb6447240c7a21612c339917d90 in term 1.
I20260806 04:52:14.098634 21993 raft_consensus.cc:2454] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 476a7eb6447240c7a21612c339917d90 in term 1.
I20260806 04:52:14.098636 21779 leader_election.cc:302] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 476a7eb6447240c7a21612c339917d90, 53d1b1fef3514a09bb73d4a956107a36; no voters:
I20260806 04:52:14.098873 22212 raft_consensus.cc:2781] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:52:14.099259 22212 raft_consensus.cc:686] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [term 1 LEADER]: Becoming Leader. State: Replica: 476a7eb6447240c7a21612c339917d90, State: Running, Role: LEADER
I20260806 04:52:14.099659 22212 consensus_queue.cc:237] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } }
I20260806 04:52:14.104043 21687 catalog_manager.cc:5495] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 476a7eb6447240c7a21612c339917d90 (127.13.148.129). New cstate: current_term: 1 leader_uuid: "476a7eb6447240c7a21612c339917d90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } health_report { overall_health: UNKNOWN } } }
W20260806 04:52:14.112157 22041 tablet.cc:2369] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:52:14.125801 21899 tablet.cc:2369] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:52:14.135044 22202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:14.130171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34646 (local address 127.13.148.129:40165)
0806 04:52:14.130376 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:14.130382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:14.130467 (+ 85us) server_negotiation.cc:407] Connection header received
0806 04:52:14.130718 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:14.130721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:14.130788 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:14.130954 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:52:14.131306 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.131982 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.133247 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.133602 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.133720 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.134128 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:52:14.134559 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:52:14.134705 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:14.134757 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 04:52:14.134823 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 04:52:14.141978 21993 raft_consensus.cc:1261] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937 [term 1 FOLLOWER]: Refusing update from remote peer 476a7eb6447240c7a21612c339917d90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:52:14.141988 22131 raft_consensus.cc:1261] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36 [term 1 FOLLOWER]: Refusing update from remote peer 476a7eb6447240c7a21612c339917d90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:52:14.142792 22212 consensus_queue.cc:1035] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:52:14.143069 22217 consensus_queue.cc:1035] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:52:14.153373 22222 mvcc.cc:205] Tried to move back new op lower bound from 7315422962237227008 to 7315422962075336704. Current Snapshot: MvccSnapshot[applied={T|T < 7315422962237227008}]
I20260806 04:52:14.165225 21687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:52:14.165493 21687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:52:14.169251 21687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6: Sending DeleteTablet for 3 replicas of tablet 40c8f1a166e54ad6bbad94e9031ab38b
I20260806 04:52:14.170228 22110 tablet_service.cc:1514] Processing DeleteTablet for tablet 40c8f1a166e54ad6bbad94e9031ab38b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:14 UTC) from {username='oryx', principal='oryx/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:49942
I20260806 04:52:14.170279 21832 tablet_service.cc:1514] Processing DeleteTablet for tablet 40c8f1a166e54ad6bbad94e9031ab38b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:14 UTC) from {username='oryx', principal='oryx/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:34644
I20260806 04:52:14.171135 21972 tablet_service.cc:1514] Processing DeleteTablet for tablet 40c8f1a166e54ad6bbad94e9031ab38b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:14 UTC) from {username='oryx', principal='oryx/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:32958
I20260806 04:52:14.171142 21689 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:52:14.171775 21689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:52:14.180315 21831 tablet_service.cc:1467] Processing CreateTablet for tablet b5a04c8fef9547ebb010266276c32348 (DEFAULT_TABLE table=test-table [id=61ac16a70a114c1db1a93d7355fce552]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:14.180961 21831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a04c8fef9547ebb010266276c32348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:14.181377 22230 tablet_replica.cc:330] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90: stopping tablet replica
I20260806 04:52:14.181931 22230 raft_consensus.cc:2229] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:52:14.182694 22230 raft_consensus.cc:2258] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:14.185552 21970 tablet_service.cc:1467] Processing CreateTablet for tablet b5a04c8fef9547ebb010266276c32348 (DEFAULT_TABLE table=test-table [id=61ac16a70a114c1db1a93d7355fce552]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:14.185892 22109 tablet_service.cc:1467] Processing CreateTablet for tablet b5a04c8fef9547ebb010266276c32348 (DEFAULT_TABLE table=test-table [id=61ac16a70a114c1db1a93d7355fce552]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:14.186236 21970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a04c8fef9547ebb010266276c32348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:14.186410 22109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a04c8fef9547ebb010266276c32348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:14.186471 22231 tablet_replica.cc:330] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937: stopping tablet replica
I20260806 04:52:14.186858 22231 raft_consensus.cc:2229] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:52:14.187336 22231 raft_consensus.cc:2258] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:14.189355 22231 ts_tablet_manager.cc:1905] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:14.192600 22210 tablet_bootstrap.cc:492] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36: Bootstrap starting.
I20260806 04:52:14.194195 22210 tablet_bootstrap.cc:654] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:14.195823 22231 ts_tablet_manager.cc:1918] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:52:14.196007 22231 log.cc:1196] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/40c8f1a166e54ad6bbad94e9031ab38b
I20260806 04:52:14.196581 22231 ts_tablet_manager.cc:1939] T 40c8f1a166e54ad6bbad94e9031ab38b P 46f4d9c56fa54125b72c2256a2e86937: Deleting consensus metadata
I20260806 04:52:14.197278 22207 tablet_bootstrap.cc:492] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937: Bootstrap starting.
I20260806 04:52:14.197532 22229 tablet_replica.cc:330] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36: stopping tablet replica
I20260806 04:52:14.198017 22229 raft_consensus.cc:2229] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:52:14.198585 22229 raft_consensus.cc:2258] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:14.198719 22207 tablet_bootstrap.cc:654] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:14.199779 22206 tablet_bootstrap.cc:492] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90: Bootstrap starting.
I20260806 04:52:14.200379 21673 catalog_manager.cc:4841] TS 46f4d9c56fa54125b72c2256a2e86937 (127.13.148.130:42645): tablet 40c8f1a166e54ad6bbad94e9031ab38b (table test-table [id=76d0083dfcfc4a1e999a330b07d6e4dd]) successfully deleted
I20260806 04:52:14.200733 22229 ts_tablet_manager.cc:1905] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:14.203558 22206 tablet_bootstrap.cc:654] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:14.204677 22210 tablet_bootstrap.cc:492] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36: No bootstrap required, opened a new log
I20260806 04:52:14.204878 22210 ts_tablet_manager.cc:1397] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260806 04:52:14.205798 22210 raft_consensus.cc:348] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.206101 22210 raft_consensus.cc:374] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:14.206252 22210 raft_consensus.cc:729] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1b1fef3514a09bb73d4a956107a36, State: Initialized, Role: FOLLOWER
I20260806 04:52:14.206609 22210 consensus_queue.cc:260] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.207211 22210 ts_tablet_manager.cc:1428] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 04:52:14.207767 22207 tablet_bootstrap.cc:492] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937: No bootstrap required, opened a new log
I20260806 04:52:14.207940 22207 ts_tablet_manager.cc:1397] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260806 04:52:14.208680 22229 ts_tablet_manager.cc:1918] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:52:14.208658 22207 raft_consensus.cc:348] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.208948 22207 raft_consensus.cc:374] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:14.209075 22207 raft_consensus.cc:729] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46f4d9c56fa54125b72c2256a2e86937, State: Initialized, Role: FOLLOWER
I20260806 04:52:14.208920 22229 log.cc:1196] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/40c8f1a166e54ad6bbad94e9031ab38b
I20260806 04:52:14.209371 22207 consensus_queue.cc:260] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.210004 22229 ts_tablet_manager.cc:1939] T 40c8f1a166e54ad6bbad94e9031ab38b P 53d1b1fef3514a09bb73d4a956107a36: Deleting consensus metadata
I20260806 04:52:14.211572 21674 catalog_manager.cc:4841] TS 53d1b1fef3514a09bb73d4a956107a36 (127.13.148.131:43631): tablet 40c8f1a166e54ad6bbad94e9031ab38b (table test-table [id=76d0083dfcfc4a1e999a330b07d6e4dd]) successfully deleted
I20260806 04:52:14.212642 22207 ts_tablet_manager.cc:1428] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260806 04:52:14.213340 22230 ts_tablet_manager.cc:1905] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:14.216154 22206 tablet_bootstrap.cc:492] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90: No bootstrap required, opened a new log
I20260806 04:52:14.216331 22206 ts_tablet_manager.cc:1397] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20260806 04:52:14.217271 22206 raft_consensus.cc:348] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.217729 22206 raft_consensus.cc:374] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:14.217888 22206 raft_consensus.cc:729] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476a7eb6447240c7a21612c339917d90, State: Initialized, Role: FOLLOWER
I20260806 04:52:14.218189 22206 consensus_queue.cc:260] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.218763 22206 ts_tablet_manager.cc:1428] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 04:52:14.220142 22230 ts_tablet_manager.cc:1918] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:52:14.220307 22230 log.cc:1196] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/40c8f1a166e54ad6bbad94e9031ab38b
I20260806 04:52:14.220952 22230 ts_tablet_manager.cc:1939] T 40c8f1a166e54ad6bbad94e9031ab38b P 476a7eb6447240c7a21612c339917d90: Deleting consensus metadata
I20260806 04:52:14.222503 21673 catalog_manager.cc:4841] TS 476a7eb6447240c7a21612c339917d90 (127.13.148.129:40165): tablet 40c8f1a166e54ad6bbad94e9031ab38b (table test-table [id=76d0083dfcfc4a1e999a330b07d6e4dd]) successfully deleted
I20260806 04:52:14.440181 22213 raft_consensus.cc:482] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:52:14.440410 22213 raft_consensus.cc:504] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.442031 22213 leader_election.cc:288] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 476a7eb6447240c7a21612c339917d90 (127.13.148.129:40165), 53d1b1fef3514a09bb73d4a956107a36 (127.13.148.131:43631)
I20260806 04:52:14.449667 22202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:14.442405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:55191 (local address 127.13.148.129:40165)
0806 04:52:14.442625 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:14.442630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:14.443790 (+ 1160us) server_negotiation.cc:407] Connection header received
0806 04:52:14.443973 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:14.443981 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:14.444034 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:14.444115 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:14.444812 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.445823 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.447908 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.448641 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.448817 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.448889 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:14.449204 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 04:52:14.449356 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 04:52:14.449827 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:14.442645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:49557 (local address 127.13.148.131:43631)
0806 04:52:14.442825 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:14.442832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:14.443198 (+ 366us) server_negotiation.cc:407] Connection header received
0806 04:52:14.443493 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:14.443499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:14.443591 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:14.443655 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:14.444434 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.445469 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.447877 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.448972 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.449109 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.449174 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:14.449496 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 04:52:14.449611 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 04:52:14.450436 22235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:14.442327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:43631 (local address 127.13.148.130:49557)
0806 04:52:14.443091 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:14.443112 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:14.443316 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:14.443869 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:14.443873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:14.443900 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:14.444223 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:14.444239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.445584 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.445588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:14.447720 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:14.447735 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.449181 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.449186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:14.449320 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.449348 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 04:52:14.449984 (+ 636us) client_negotiation.cc:240] Negotiation successful
0806 04:52:14.450074 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":5,"thread_start_us":537,"threads_started":1}
I20260806 04:52:14.450416 22236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:14.442612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:40165 (local address 127.13.148.130:55191)
0806 04:52:14.443695 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:14.443707 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:14.443839 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:14.444145 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:14.444148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:14.444174 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:14.444621 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:14.444633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.445963 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.445967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:14.447757 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:14.447767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.448855 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.448861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:14.449009 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.449048 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 04:52:14.449830 (+ 782us) client_negotiation.cc:240] Negotiation successful
0806 04:52:14.450026 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":1,"thread_start_us":797,"threads_started":1}
I20260806 04:52:14.451102 22131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a04c8fef9547ebb010266276c32348" candidate_uuid: "46f4d9c56fa54125b72c2256a2e86937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53d1b1fef3514a09bb73d4a956107a36" is_pre_election: true
I20260806 04:52:14.451387 22131 raft_consensus.cc:2454] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46f4d9c56fa54125b72c2256a2e86937 in term 0.
I20260806 04:52:14.451426 21852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a04c8fef9547ebb010266276c32348" candidate_uuid: "46f4d9c56fa54125b72c2256a2e86937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476a7eb6447240c7a21612c339917d90" is_pre_election: true
I20260806 04:52:14.451889 21921 leader_election.cc:302] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46f4d9c56fa54125b72c2256a2e86937, 53d1b1fef3514a09bb73d4a956107a36; no voters:
I20260806 04:52:14.451897 21852 raft_consensus.cc:2454] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46f4d9c56fa54125b72c2256a2e86937 in term 0.
I20260806 04:52:14.452303 22213 raft_consensus.cc:2781] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:52:14.452416 22213 raft_consensus.cc:482] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:52:14.452524 22213 raft_consensus.cc:3037] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:14.455562 22213 raft_consensus.cc:504] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.456192 22213 leader_election.cc:288] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [CANDIDATE]: Term 1 election: Requested vote from peers 476a7eb6447240c7a21612c339917d90 (127.13.148.129:40165), 53d1b1fef3514a09bb73d4a956107a36 (127.13.148.131:43631)
I20260806 04:52:14.456602 22131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a04c8fef9547ebb010266276c32348" candidate_uuid: "46f4d9c56fa54125b72c2256a2e86937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53d1b1fef3514a09bb73d4a956107a36"
I20260806 04:52:14.456627 21852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a04c8fef9547ebb010266276c32348" candidate_uuid: "46f4d9c56fa54125b72c2256a2e86937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476a7eb6447240c7a21612c339917d90"
I20260806 04:52:14.456832 22131 raft_consensus.cc:3037] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:14.456866 21852 raft_consensus.cc:3037] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:14.458684 22131 raft_consensus.cc:2454] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46f4d9c56fa54125b72c2256a2e86937 in term 1.
I20260806 04:52:14.459025 21921 leader_election.cc:302] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46f4d9c56fa54125b72c2256a2e86937, 53d1b1fef3514a09bb73d4a956107a36; no voters:
I20260806 04:52:14.459287 22213 raft_consensus.cc:2781] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:52:14.459760 22213 raft_consensus.cc:686] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [term 1 LEADER]: Becoming Leader. State: Replica: 46f4d9c56fa54125b72c2256a2e86937, State: Running, Role: LEADER
I20260806 04:52:14.460160 22213 consensus_queue.cc:237] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.460451 21852 raft_consensus.cc:2454] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46f4d9c56fa54125b72c2256a2e86937 in term 1.
I20260806 04:52:14.463851 21689 catalog_manager.cc:5495] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46f4d9c56fa54125b72c2256a2e86937 (127.13.148.130). New cstate: current_term: 1 leader_uuid: "46f4d9c56fa54125b72c2256a2e86937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } health_report { overall_health: UNKNOWN } } }
Aug 06 04:52:14 dist-test-slave-1x4g krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991932, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.190@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM
I20260806 04:52:14.531944 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:14.519036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:45859 (local address 127.0.0.1:32978)
0806 04:52:14.519850 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:14.519863 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:14.520007 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:14.520294 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:14.520300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:14.521062 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:14.521250 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:14.521259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.522355 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.522359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:14.523160 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:14.523212 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.523923 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.523928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:14.524080 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.524711 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:14.524726 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:14.526642 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:14.529428 (+ 2786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:14.529440 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:14.529447 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:14.529836 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:14.530165 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:14.530171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:14.530174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:14.530289 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:14.530805 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:14.530812 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:14.530993 (+ 181us) client_negotiation.cc:769] Sending connection context
0806 04:52:14.531178 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 04:52:14.531470 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"spinlock_wait_cycles":8576,"thread_start_us":587,"threads_started":1}
I20260806 04:52:14.531965 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:14.519045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32978 (local address 127.13.148.190:45859)
0806 04:52:14.519729 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:14.519736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:14.519943 (+ 207us) server_negotiation.cc:407] Connection header received
0806 04:52:14.520117 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:14.520121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:14.520166 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:14.520260 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:14.521398 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.522217 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.523333 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.523777 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.523890 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.526802 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:14.526806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:14.526809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:14.526833 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:14.529322 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:14.529943 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:14.529958 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:14.529965 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:14.530069 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:14.530382 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:14.530389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:14.530392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:14.530666 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:14.530827 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:14.531273 (+ 446us) server_negotiation.cc:299] Negotiation successful
0806 04:52:14.531582 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":515,"thread_start_us":340,"threads_started":1}
I20260806 04:52:14.540516 21689 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:32978:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 04:52:14.541471 21689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:52:14.547617 21831 tablet_service.cc:1467] Processing CreateTablet for tablet 0dac69a9456b4b019b47254105fc10de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d5880750ff584217954113b49aee4341]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:52:14.547847 21970 tablet_service.cc:1467] Processing CreateTablet for tablet 0dac69a9456b4b019b47254105fc10de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d5880750ff584217954113b49aee4341]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:52:14.548141 21831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dac69a9456b4b019b47254105fc10de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:14.548386 21970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dac69a9456b4b019b47254105fc10de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:14.549043 22110 tablet_service.cc:1467] Processing CreateTablet for tablet 0dac69a9456b4b019b47254105fc10de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d5880750ff584217954113b49aee4341]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:52:14.549512 22110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dac69a9456b4b019b47254105fc10de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:14.554138 22206 tablet_bootstrap.cc:492] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90: Bootstrap starting.
I20260806 04:52:14.554746 22207 tablet_bootstrap.cc:492] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937: Bootstrap starting.
I20260806 04:52:14.555620 22210 tablet_bootstrap.cc:492] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36: Bootstrap starting.
I20260806 04:52:14.555666 22206 tablet_bootstrap.cc:654] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:14.556267 22207 tablet_bootstrap.cc:654] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:14.557288 22210 tablet_bootstrap.cc:654] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:14.558326 22206 tablet_bootstrap.cc:492] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90: No bootstrap required, opened a new log
I20260806 04:52:14.558485 22206 ts_tablet_manager.cc:1397] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260806 04:52:14.559139 22210 tablet_bootstrap.cc:492] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36: No bootstrap required, opened a new log
I20260806 04:52:14.559049 22206 raft_consensus.cc:348] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.559264 22206 raft_consensus.cc:374] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:14.559294 22210 ts_tablet_manager.cc:1397] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260806 04:52:14.559347 22206 raft_consensus.cc:729] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476a7eb6447240c7a21612c339917d90, State: Initialized, Role: FOLLOWER
I20260806 04:52:14.559573 22207 tablet_bootstrap.cc:492] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937: No bootstrap required, opened a new log
I20260806 04:52:14.559705 22207 ts_tablet_manager.cc:1397] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 04:52:14.559583 22206 consensus_queue.cc:260] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.560151 22206 ts_tablet_manager.cc:1428] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260806 04:52:14.560129 22210 raft_consensus.cc:348] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.560341 22210 raft_consensus.cc:374] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:14.560411 22210 raft_consensus.cc:729] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1b1fef3514a09bb73d4a956107a36, State: Initialized, Role: FOLLOWER
I20260806 04:52:14.560381 22207 raft_consensus.cc:348] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.560426 22212 tablet_replica.cc:439] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } }
I20260806 04:52:14.560604 22207 raft_consensus.cc:374] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:14.560644 22212 tablet_replica.cc:442] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:52:14.560673 22207 raft_consensus.cc:729] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46f4d9c56fa54125b72c2256a2e86937, State: Initialized, Role: FOLLOWER
I20260806 04:52:14.560612 22210 consensus_queue.cc:260] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.560913 22207 consensus_queue.cc:260] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.561095 22214 tablet_replica.cc:439] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } }
I20260806 04:52:14.561378 22214 tablet_replica.cc:442] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:52:14.561496 22207 ts_tablet_manager.cc:1428] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 04:52:14.561635 22213 tablet_replica.cc:439] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } }
I20260806 04:52:14.561896 22213 tablet_replica.cc:442] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:52:14.562240 22210 ts_tablet_manager.cc:1428] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260806 04:52:14.659318 22213 raft_consensus.cc:482] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:52:14.659538 22213 raft_consensus.cc:504] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.660106 22213 leader_election.cc:288] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 476a7eb6447240c7a21612c339917d90 (127.13.148.129:40165), 53d1b1fef3514a09bb73d4a956107a36 (127.13.148.131:43631)
I20260806 04:52:14.660593 21852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dac69a9456b4b019b47254105fc10de" candidate_uuid: "46f4d9c56fa54125b72c2256a2e86937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476a7eb6447240c7a21612c339917d90" is_pre_election: true
I20260806 04:52:14.660626 22131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dac69a9456b4b019b47254105fc10de" candidate_uuid: "46f4d9c56fa54125b72c2256a2e86937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53d1b1fef3514a09bb73d4a956107a36" is_pre_election: true
I20260806 04:52:14.660861 21852 raft_consensus.cc:2454] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46f4d9c56fa54125b72c2256a2e86937 in term 0.
I20260806 04:52:14.660982 22131 raft_consensus.cc:2454] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46f4d9c56fa54125b72c2256a2e86937 in term 0.
I20260806 04:52:14.661319 21920 leader_election.cc:302] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46f4d9c56fa54125b72c2256a2e86937, 476a7eb6447240c7a21612c339917d90; no voters:
I20260806 04:52:14.661619 22213 raft_consensus.cc:2781] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:52:14.661718 22213 raft_consensus.cc:482] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:52:14.661818 22213 raft_consensus.cc:3037] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:14.663689 22213 raft_consensus.cc:504] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.664201 22213 leader_election.cc:288] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [CANDIDATE]: Term 1 election: Requested vote from peers 476a7eb6447240c7a21612c339917d90 (127.13.148.129:40165), 53d1b1fef3514a09bb73d4a956107a36 (127.13.148.131:43631)
I20260806 04:52:14.664508 21852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dac69a9456b4b019b47254105fc10de" candidate_uuid: "46f4d9c56fa54125b72c2256a2e86937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476a7eb6447240c7a21612c339917d90"
I20260806 04:52:14.664737 21852 raft_consensus.cc:3037] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:14.664705 22131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dac69a9456b4b019b47254105fc10de" candidate_uuid: "46f4d9c56fa54125b72c2256a2e86937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53d1b1fef3514a09bb73d4a956107a36"
I20260806 04:52:14.664921 22131 raft_consensus.cc:3037] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:14.667315 21852 raft_consensus.cc:2454] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46f4d9c56fa54125b72c2256a2e86937 in term 1.
I20260806 04:52:14.667461 22131 raft_consensus.cc:2454] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46f4d9c56fa54125b72c2256a2e86937 in term 1.
I20260806 04:52:14.667654 21920 leader_election.cc:302] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46f4d9c56fa54125b72c2256a2e86937, 476a7eb6447240c7a21612c339917d90; no voters:
I20260806 04:52:14.667922 22213 raft_consensus.cc:2781] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:52:14.668176 22213 raft_consensus.cc:686] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [term 1 LEADER]: Becoming Leader. State: Replica: 46f4d9c56fa54125b72c2256a2e86937, State: Running, Role: LEADER
I20260806 04:52:14.668433 22213 consensus_queue.cc:237] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.669857 22237 tablet_replica.cc:439] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937: TxnStatusTablet state changed. Reason: New leader 46f4d9c56fa54125b72c2256a2e86937. Latest consensus state: current_term: 1 leader_uuid: "46f4d9c56fa54125b72c2256a2e86937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } }
I20260806 04:52:14.670123 22237 tablet_replica.cc:442] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:52:14.670740 22250 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 04:52:14.671011 22250 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 04:52:14.671475 21688 catalog_manager.cc:5495] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46f4d9c56fa54125b72c2256a2e86937 (127.13.148.130). New cstate: current_term: 1 leader_uuid: "46f4d9c56fa54125b72c2256a2e86937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } health_report { overall_health: UNKNOWN } } }
I20260806 04:52:14.714437 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:14.708432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:42645 (local address 127.0.0.1:32972)
0806 04:52:14.708712 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:14.708736 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:14.708899 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:14.709479 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:14.709481 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:14.709491 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:52:14.709632 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:14.709638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.710984 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.710987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:14.711954 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:14.711965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.712877 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.712881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:14.713108 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.713169 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:52:14.714073 (+ 904us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:52:14.714092 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:52:14.714138 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 04:52:14.714233 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 04:52:14.714529 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:14.708394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32972 (local address 127.13.148.130:42645)
0806 04:52:14.709055 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:14.709061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:14.709080 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:52:14.709184 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:14.709189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:14.709298 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:14.709463 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:52:14.709755 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.710862 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.712100 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.712715 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.712852 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.713344 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:52:14.713976 (+ 632us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:52:14.714111 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:14.714164 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 04:52:14.714234 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":355,"threads_started":1}
I20260806 04:52:14.718601 22131 raft_consensus.cc:1261] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36 [term 1 FOLLOWER]: Refusing update from remote peer 46f4d9c56fa54125b72c2256a2e86937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:52:14.718706 21852 raft_consensus.cc:1261] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90 [term 1 FOLLOWER]: Refusing update from remote peer 46f4d9c56fa54125b72c2256a2e86937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:52:14.719226 22237 consensus_queue.cc:1035] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:52:14.719512 22213 consensus_queue.cc:1035] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:52:14.729985 22214 tablet_replica.cc:439] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36: TxnStatusTablet state changed. Reason: New leader 46f4d9c56fa54125b72c2256a2e86937. Latest consensus state: current_term: 1 leader_uuid: "46f4d9c56fa54125b72c2256a2e86937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } }
I20260806 04:52:14.730243 22214 tablet_replica.cc:442] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:52:14.731119 22212 tablet_replica.cc:439] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90: TxnStatusTablet state changed. Reason: New leader 46f4d9c56fa54125b72c2256a2e86937. Latest consensus state: current_term: 1 leader_uuid: "46f4d9c56fa54125b72c2256a2e86937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } }
I20260806 04:52:14.731384 22212 tablet_replica.cc:442] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:52:14.732951 22237 tablet_replica.cc:439] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46f4d9c56fa54125b72c2256a2e86937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } }
I20260806 04:52:14.733203 22237 tablet_replica.cc:442] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:52:14.734331 22255 mvcc.cc:205] Tried to move back new op lower bound from 7315422964600373248 to 7315422964404088832. Current Snapshot: MvccSnapshot[applied={T|T < 7315422964600373248}]
I20260806 04:52:14.734304 22213 tablet_replica.cc:439] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46f4d9c56fa54125b72c2256a2e86937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } }
I20260806 04:52:14.734542 22213 tablet_replica.cc:442] T 0dac69a9456b4b019b47254105fc10de P 46f4d9c56fa54125b72c2256a2e86937: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:52:14.739758 22214 tablet_replica.cc:439] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46f4d9c56fa54125b72c2256a2e86937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } }
I20260806 04:52:14.739976 22214 tablet_replica.cc:442] T 0dac69a9456b4b019b47254105fc10de P 53d1b1fef3514a09bb73d4a956107a36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:52:14.744755 22212 tablet_replica.cc:439] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46f4d9c56fa54125b72c2256a2e86937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } } }
I20260806 04:52:14.745731 22212 tablet_replica.cc:442] T 0dac69a9456b4b019b47254105fc10de P 476a7eb6447240c7a21612c339917d90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:52:14.749775 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:14.744471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32986 (local address 127.13.148.130:42645)
0806 04:52:14.744616 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:14.744621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:14.745296 (+ 675us) server_negotiation.cc:407] Connection header received
0806 04:52:14.745479 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:14.745486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:14.745569 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:14.745656 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:52:14.746306 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.747111 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.748290 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.748681 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.748832 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.749172 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:52:14.749256 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:52:14.749353 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:14.749458 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 04:52:14.749529 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:52:14.764433 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:14.758979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:36233 (local address 127.13.148.130:42645)
0806 04:52:14.759136 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:14.759141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:14.759810 (+ 669us) server_negotiation.cc:407] Connection header received
0806 04:52:14.759987 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:14.759991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:14.760060 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:14.760154 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:52:14.760744 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.761534 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.762839 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.763212 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.763354 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.763736 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:52:14.763823 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:52:14.763913 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:14.764052 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 04:52:14.764106 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":38}
I20260806 04:52:14.764532 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:14.759120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:42645 (local address 127.13.148.130:36233)
0806 04:52:14.759693 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:14.759707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:14.759874 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:14.760169 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:14.760173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:14.760182 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:52:14.760627 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:14.760635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.761655 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.761662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:14.762720 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:14.762731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.763392 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.763400 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:14.763528 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.763574 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:52:14.763914 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:52:14.763925 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 04:52:14.764037 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 04:52:14.764151 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":4,"thread_start_us":316,"threads_started":1}
I20260806 04:52:14.775097 21852 raft_consensus.cc:1261] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90 [term 1 FOLLOWER]: Refusing update from remote peer 46f4d9c56fa54125b72c2256a2e86937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:52:14.775599 22237 consensus_queue.cc:1035] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:52:14.778813 22131 raft_consensus.cc:1261] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36 [term 1 FOLLOWER]: Refusing update from remote peer 46f4d9c56fa54125b72c2256a2e86937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:52:14.779364 22213 consensus_queue.cc:1035] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:52:14.836477 21688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:52:14.836769 21688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:52:14.839922 21688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6: Sending DeleteTablet for 3 replicas of tablet b5a04c8fef9547ebb010266276c32348
I20260806 04:52:14.840585 21831 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a04c8fef9547ebb010266276c32348 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:14 UTC) from {username='oryx', principal='oryx/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:34644
I20260806 04:52:14.840906 22110 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a04c8fef9547ebb010266276c32348 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:14 UTC) from {username='oryx', principal='oryx/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:49942
I20260806 04:52:14.845682 21970 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a04c8fef9547ebb010266276c32348 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:14 UTC) from {username='oryx', principal='oryx/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:32958
I20260806 04:52:14.851603 22273 tablet_replica.cc:330] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36: stopping tablet replica
I20260806 04:52:14.851900 22273 raft_consensus.cc:2229] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:52:14.852231 22273 raft_consensus.cc:2258] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:14.853317 22273 ts_tablet_manager.cc:1905] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:14.860690 22273 ts_tablet_manager.cc:1918] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:52:14.860910 22273 log.cc:1196] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/b5a04c8fef9547ebb010266276c32348
I20260806 04:52:14.861555 22273 ts_tablet_manager.cc:1939] T b5a04c8fef9547ebb010266276c32348 P 53d1b1fef3514a09bb73d4a956107a36: Deleting consensus metadata
I20260806 04:52:14.862669 21674 catalog_manager.cc:4841] TS 53d1b1fef3514a09bb73d4a956107a36 (127.13.148.131:43631): tablet b5a04c8fef9547ebb010266276c32348 (table test-table [id=61ac16a70a114c1db1a93d7355fce552]) successfully deleted
I20260806 04:52:14.865360 22274 tablet_replica.cc:330] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937: stopping tablet replica
I20260806 04:52:14.865845 22272 tablet_replica.cc:330] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90: stopping tablet replica
I20260806 04:52:14.866113 22272 raft_consensus.cc:2229] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:52:14.866379 22274 raft_consensus.cc:2229] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:52:14.866699 22272 raft_consensus.cc:2258] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:14.867328 22274 raft_consensus.cc:2258] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:14.869293 22274 ts_tablet_manager.cc:1905] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:14.869410 22272 ts_tablet_manager.cc:1905] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:14.882022 22274 ts_tablet_manager.cc:1918] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:52:14.883538 22272 ts_tablet_manager.cc:1918] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:52:14.883725 22274 log.cc:1196] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/b5a04c8fef9547ebb010266276c32348
I20260806 04:52:14.883872 22272 log.cc:1196] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/b5a04c8fef9547ebb010266276c32348
I20260806 04:52:14.884303 22274 ts_tablet_manager.cc:1939] T b5a04c8fef9547ebb010266276c32348 P 46f4d9c56fa54125b72c2256a2e86937: Deleting consensus metadata
I20260806 04:52:14.884727 22272 ts_tablet_manager.cc:1939] T b5a04c8fef9547ebb010266276c32348 P 476a7eb6447240c7a21612c339917d90: Deleting consensus metadata
I20260806 04:52:14.885866 21673 catalog_manager.cc:4841] TS 46f4d9c56fa54125b72c2256a2e86937 (127.13.148.130:42645): tablet b5a04c8fef9547ebb010266276c32348 (table test-table [id=61ac16a70a114c1db1a93d7355fce552]) successfully deleted
I20260806 04:52:14.886514 21673 catalog_manager.cc:4841] TS 476a7eb6447240c7a21612c339917d90 (127.13.148.129:40165): tablet b5a04c8fef9547ebb010266276c32348 (table test-table [id=61ac16a70a114c1db1a93d7355fce552]) successfully deleted
Aug 06 04:52:14 dist-test-slave-1x4g krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 04:52:14 dist-test-slave-1x4g krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:14 dist-test-slave-1x4g krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 04:52:14.914921 22244 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:14.905396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32992 (local address 127.13.148.190:45859)
0806 04:52:14.905593 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:14.905599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:14.906056 (+ 457us) server_negotiation.cc:407] Connection header received
0806 04:52:14.906243 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:14.906248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:14.906321 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:14.906416 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:14.907619 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.908311 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.909518 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.909932 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.910065 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.914692 (+ 4627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32992: BlockingRecv error: recv got EOF from 127.0.0.1:32992 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 04:52:14 dist-test-slave-1x4g krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM
I20260806 04:52:14.930596 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:14.917275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.13.148.190:45859)
0806 04:52:14.917412 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:14.917417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:14.917830 (+ 413us) server_negotiation.cc:407] Connection header received
0806 04:52:14.918002 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:14.918007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:14.918071 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:14.918161 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:14.919173 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.919805 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.920914 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:14.921366 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:14.921518 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:14.925182 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:14.925188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:14.925193 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:14.925249 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:14.928342 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:14.928888 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:14.928893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:14.928896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:14.929038 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:14.929349 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:14.929354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:14.929357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:14.929675 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:14.929925 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:14.930053 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 04:52:14.930263 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:52:14.933923 21688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:52:14.934517 21688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:52:14.940750 21831 tablet_service.cc:1467] Processing CreateTablet for tablet df1e6940eeb3455f80007ef694520f5c (DEFAULT_TABLE table=test-table [id=027d0cc7134d427698db0b41bfc0a88f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:14.940973 22110 tablet_service.cc:1467] Processing CreateTablet for tablet df1e6940eeb3455f80007ef694520f5c (DEFAULT_TABLE table=test-table [id=027d0cc7134d427698db0b41bfc0a88f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:14.941324 21831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df1e6940eeb3455f80007ef694520f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:14.941404 22110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df1e6940eeb3455f80007ef694520f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:14.945174 21970 tablet_service.cc:1467] Processing CreateTablet for tablet df1e6940eeb3455f80007ef694520f5c (DEFAULT_TABLE table=test-table [id=027d0cc7134d427698db0b41bfc0a88f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:14.945623 21970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df1e6940eeb3455f80007ef694520f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:14.946132 22206 tablet_bootstrap.cc:492] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90: Bootstrap starting.
I20260806 04:52:14.947494 22210 tablet_bootstrap.cc:492] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36: Bootstrap starting.
I20260806 04:52:14.947665 22206 tablet_bootstrap.cc:654] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:14.949059 22210 tablet_bootstrap.cc:654] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:14.953882 22207 tablet_bootstrap.cc:492] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937: Bootstrap starting.
I20260806 04:52:14.955628 22207 tablet_bootstrap.cc:654] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:14.959000 22210 tablet_bootstrap.cc:492] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36: No bootstrap required, opened a new log
I20260806 04:52:14.959133 22210 ts_tablet_manager.cc:1397] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260806 04:52:14.960000 22210 raft_consensus.cc:348] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.960250 22210 raft_consensus.cc:374] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:14.960402 22210 raft_consensus.cc:729] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1b1fef3514a09bb73d4a956107a36, State: Initialized, Role: FOLLOWER
I20260806 04:52:14.960853 22206 tablet_bootstrap.cc:492] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90: No bootstrap required, opened a new log
I20260806 04:52:14.960716 22210 consensus_queue.cc:260] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.961038 22206 ts_tablet_manager.cc:1397] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20260806 04:52:14.961362 22210 ts_tablet_manager.cc:1428] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 04:52:14.962883 22206 raft_consensus.cc:348] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.963133 22206 raft_consensus.cc:374] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:14.963227 22206 raft_consensus.cc:729] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476a7eb6447240c7a21612c339917d90, State: Initialized, Role: FOLLOWER
I20260806 04:52:14.963438 22206 consensus_queue.cc:260] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.964052 22206 ts_tablet_manager.cc:1428] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:52:14.964870 22207 tablet_bootstrap.cc:492] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937: No bootstrap required, opened a new log
I20260806 04:52:14.965010 22207 ts_tablet_manager.cc:1397] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260806 04:52:14.965735 22207 raft_consensus.cc:348] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.965945 22207 raft_consensus.cc:374] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:14.966059 22207 raft_consensus.cc:729] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46f4d9c56fa54125b72c2256a2e86937, State: Initialized, Role: FOLLOWER
I20260806 04:52:14.966286 22207 consensus_queue.cc:260] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:14.966902 22207 ts_tablet_manager.cc:1428] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 04:52:15.227682 22214 raft_consensus.cc:482] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:52:15.227957 22214 raft_consensus.cc:504] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:15.229744 22214 leader_election.cc:288] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 476a7eb6447240c7a21612c339917d90 (127.13.148.129:40165), 46f4d9c56fa54125b72c2256a2e86937 (127.13.148.130:42645)
I20260806 04:52:15.239758 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:15.230060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:39303 (local address 127.13.148.129:40165)
0806 04:52:15.233158 (+ 3098us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:15.233164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:15.233428 (+ 264us) server_negotiation.cc:407] Connection header received
0806 04:52:15.233664 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:15.233669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:15.233762 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:15.234113 (+ 351us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:15.234689 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:15.235753 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:15.237497 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:15.238734 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:15.238904 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:15.238994 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:15.239272 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 04:52:15.239430 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2977,"thread_start_us":2877,"threads_started":1}
I20260806 04:52:15.239785 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:15.230056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:40165 (local address 127.13.148.131:39303)
0806 04:52:15.233319 (+ 3263us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:15.233336 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:15.233476 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:15.234147 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:15.234152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:15.234189 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:15.234471 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:15.234486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:15.235949 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:15.235952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:15.237302 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:15.237321 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:15.238918 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:15.238924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:15.239091 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:15.239128 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 04:52:15.239233 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 04:52:15.239374 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3168,"thread_start_us":3074,"threads_started":1}
I20260806 04:52:15.240608 21852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1e6940eeb3455f80007ef694520f5c" candidate_uuid: "53d1b1fef3514a09bb73d4a956107a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476a7eb6447240c7a21612c339917d90" is_pre_election: true
I20260806 04:52:15.240891 21852 raft_consensus.cc:2454] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53d1b1fef3514a09bb73d4a956107a36 in term 0.
I20260806 04:52:15.241142 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:15.233730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:50497 (local address 127.13.148.130:42645)
0806 04:52:15.233937 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:15.233963 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:15.234861 (+ 898us) server_negotiation.cc:407] Connection header received
0806 04:52:15.235074 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:15.235079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:15.235168 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:15.235228 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:15.235913 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:15.237048 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:15.238736 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:15.240134 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:15.240272 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:15.240331 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:15.240694 (+ 363us) server_negotiation.cc:299] Negotiation successful
0806 04:52:15.240857 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:52:15.241856 22293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:15.233618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:42645 (local address 127.13.148.131:50497)
0806 04:52:15.234758 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:15.234777 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:15.234887 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:15.235397 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:15.235402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:15.235433 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:15.235740 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:15.235750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:15.237201 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:15.237206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:15.238577 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:15.238589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:15.240309 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:15.240315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:15.240520 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:15.240551 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 04:52:15.241357 (+ 806us) client_negotiation.cc:240] Negotiation successful
0806 04:52:15.241505 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"thread_start_us":828,"threads_started":1}
I20260806 04:52:15.241731 22061 leader_election.cc:302] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 476a7eb6447240c7a21612c339917d90, 53d1b1fef3514a09bb73d4a956107a36; no voters:
I20260806 04:52:15.242244 22214 raft_consensus.cc:2781] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:52:15.242385 22214 raft_consensus.cc:482] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:52:15.242465 22214 raft_consensus.cc:3037] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:15.242666 21993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1e6940eeb3455f80007ef694520f5c" candidate_uuid: "53d1b1fef3514a09bb73d4a956107a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46f4d9c56fa54125b72c2256a2e86937" is_pre_election: true
I20260806 04:52:15.242877 21993 raft_consensus.cc:2454] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53d1b1fef3514a09bb73d4a956107a36 in term 0.
I20260806 04:52:15.245267 22214 raft_consensus.cc:504] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:15.245846 22214 leader_election.cc:288] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [CANDIDATE]: Term 1 election: Requested vote from peers 476a7eb6447240c7a21612c339917d90 (127.13.148.129:40165), 46f4d9c56fa54125b72c2256a2e86937 (127.13.148.130:42645)
I20260806 04:52:15.246248 21852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1e6940eeb3455f80007ef694520f5c" candidate_uuid: "53d1b1fef3514a09bb73d4a956107a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476a7eb6447240c7a21612c339917d90"
I20260806 04:52:15.246443 21852 raft_consensus.cc:3037] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:15.246487 21993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1e6940eeb3455f80007ef694520f5c" candidate_uuid: "53d1b1fef3514a09bb73d4a956107a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46f4d9c56fa54125b72c2256a2e86937"
I20260806 04:52:15.246655 21993 raft_consensus.cc:3037] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:15.248580 21993 raft_consensus.cc:2454] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53d1b1fef3514a09bb73d4a956107a36 in term 1.
I20260806 04:52:15.248974 22061 leader_election.cc:302] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46f4d9c56fa54125b72c2256a2e86937, 53d1b1fef3514a09bb73d4a956107a36; no voters:
I20260806 04:52:15.249115 21852 raft_consensus.cc:2454] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53d1b1fef3514a09bb73d4a956107a36 in term 1.
I20260806 04:52:15.249231 22214 raft_consensus.cc:2781] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:52:15.254738 22214 raft_consensus.cc:686] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [term 1 LEADER]: Becoming Leader. State: Replica: 53d1b1fef3514a09bb73d4a956107a36, State: Running, Role: LEADER
I20260806 04:52:15.255132 22214 consensus_queue.cc:237] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:15.258664 21688 catalog_manager.cc:5495] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53d1b1fef3514a09bb73d4a956107a36 (127.13.148.131). New cstate: current_term: 1 leader_uuid: "53d1b1fef3514a09bb73d4a956107a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } health_report { overall_health: HEALTHY } } }
I20260806 04:52:15.323918 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:15.317973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49950 (local address 127.13.148.131:43631)
0806 04:52:15.318500 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:15.318504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:15.318521 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:52:15.318597 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:15.318599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:15.318663 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:15.318821 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:52:15.319253 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:15.319980 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:15.321319 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:15.321750 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:15.321919 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:15.322587 (+ 668us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:52:15.323209 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:52:15.323349 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:15.323408 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 04:52:15.323487 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":307,"threads_started":1}
I20260806 04:52:15.328132 21993 raft_consensus.cc:1261] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937 [term 1 FOLLOWER]: Refusing update from remote peer 53d1b1fef3514a09bb73d4a956107a36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:52:15.328202 21852 raft_consensus.cc:1261] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90 [term 1 FOLLOWER]: Refusing update from remote peer 53d1b1fef3514a09bb73d4a956107a36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:52:15.328882 22214 consensus_queue.cc:1035] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:52:15.329134 22294 consensus_queue.cc:1035] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:52:15.379788 21688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:52:15.380052 21688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:52:15.383848 21688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6: Sending DeleteTablet for 3 replicas of tablet df1e6940eeb3455f80007ef694520f5c
I20260806 04:52:15.384907 22110 tablet_service.cc:1514] Processing DeleteTablet for tablet df1e6940eeb3455f80007ef694520f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:15 UTC) from {username='oryx', principal='oryx/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:49942
I20260806 04:52:15.384923 21831 tablet_service.cc:1514] Processing DeleteTablet for tablet df1e6940eeb3455f80007ef694520f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:15 UTC) from {username='oryx', principal='oryx/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:34644
I20260806 04:52:15.385257 21970 tablet_service.cc:1514] Processing DeleteTablet for tablet df1e6940eeb3455f80007ef694520f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:15 UTC) from {username='oryx', principal='oryx/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:32958
I20260806 04:52:15.385504 22274 tablet_replica.cc:330] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937: stopping tablet replica
I20260806 04:52:15.385515 22272 tablet_replica.cc:330] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90: stopping tablet replica
I20260806 04:52:15.385689 22274 raft_consensus.cc:2229] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:52:15.385684 22272 raft_consensus.cc:2229] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:52:15.385890 22274 raft_consensus.cc:2258] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:15.385892 22272 raft_consensus.cc:2258] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:15.387362 22272 ts_tablet_manager.cc:1905] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:15.387400 21688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:52:15.388053 21688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:52:15.389642 22274 ts_tablet_manager.cc:1905] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:15.397650 22307 tablet_replica.cc:330] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36: stopping tablet replica
I20260806 04:52:15.398226 22272 ts_tablet_manager.cc:1918] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:52:15.398381 22272 log.cc:1196] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/df1e6940eeb3455f80007ef694520f5c
I20260806 04:52:15.398788 22272 ts_tablet_manager.cc:1939] T df1e6940eeb3455f80007ef694520f5c P 476a7eb6447240c7a21612c339917d90: Deleting consensus metadata
I20260806 04:52:15.399381 22307 raft_consensus.cc:2229] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:52:15.399931 21673 catalog_manager.cc:4841] TS 476a7eb6447240c7a21612c339917d90 (127.13.148.129:40165): tablet df1e6940eeb3455f80007ef694520f5c (table test-table [id=027d0cc7134d427698db0b41bfc0a88f]) successfully deleted
I20260806 04:52:15.400200 21831 tablet_service.cc:1467] Processing CreateTablet for tablet d479a9e45bdb4c99bcbba0bb13ab5346 (DEFAULT_TABLE table=test-table [id=a74df609e7054e07883ccae8a2013972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:15.400545 21831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d479a9e45bdb4c99bcbba0bb13ab5346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:15.401013 22110 tablet_service.cc:1467] Processing CreateTablet for tablet d479a9e45bdb4c99bcbba0bb13ab5346 (DEFAULT_TABLE table=test-table [id=a74df609e7054e07883ccae8a2013972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:15.401471 22110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d479a9e45bdb4c99bcbba0bb13ab5346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:15.403916 22274 ts_tablet_manager.cc:1918] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:52:15.404069 22274 log.cc:1196] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/df1e6940eeb3455f80007ef694520f5c
I20260806 04:52:15.404241 21970 tablet_service.cc:1467] Processing CreateTablet for tablet d479a9e45bdb4c99bcbba0bb13ab5346 (DEFAULT_TABLE table=test-table [id=a74df609e7054e07883ccae8a2013972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:15.404449 22274 ts_tablet_manager.cc:1939] T df1e6940eeb3455f80007ef694520f5c P 46f4d9c56fa54125b72c2256a2e86937: Deleting consensus metadata
I20260806 04:52:15.404610 21970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d479a9e45bdb4c99bcbba0bb13ab5346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:15.405687 21673 catalog_manager.cc:4841] TS 46f4d9c56fa54125b72c2256a2e86937 (127.13.148.130:42645): tablet df1e6940eeb3455f80007ef694520f5c (table test-table [id=027d0cc7134d427698db0b41bfc0a88f]) successfully deleted
I20260806 04:52:15.408987 22210 tablet_bootstrap.cc:492] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36: Bootstrap starting.
I20260806 04:52:15.410720 22210 tablet_bootstrap.cc:654] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:15.415649 22206 tablet_bootstrap.cc:492] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90: Bootstrap starting.
I20260806 04:52:15.418033 22206 tablet_bootstrap.cc:654] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:15.421556 22207 tablet_bootstrap.cc:492] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937: Bootstrap starting.
I20260806 04:52:15.423939 22207 tablet_bootstrap.cc:654] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:15.427217 22206 tablet_bootstrap.cc:492] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90: No bootstrap required, opened a new log
I20260806 04:52:15.427457 22206 ts_tablet_manager.cc:1397] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260806 04:52:15.428387 22206 raft_consensus.cc:348] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:15.428813 22206 raft_consensus.cc:374] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:15.428943 22206 raft_consensus.cc:729] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476a7eb6447240c7a21612c339917d90, State: Initialized, Role: FOLLOWER
I20260806 04:52:15.429317 22307 raft_consensus.cc:2258] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:15.429270 22206 consensus_queue.cc:260] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:15.429827 22206 ts_tablet_manager.cc:1428] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260806 04:52:15.442101 22207 tablet_bootstrap.cc:492] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937: No bootstrap required, opened a new log
I20260806 04:52:15.442369 22207 ts_tablet_manager.cc:1397] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.001s
I20260806 04:52:15.443343 22207 raft_consensus.cc:348] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:15.443711 22207 raft_consensus.cc:374] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:15.443863 22207 raft_consensus.cc:729] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46f4d9c56fa54125b72c2256a2e86937, State: Initialized, Role: FOLLOWER
I20260806 04:52:15.445551 22207 consensus_queue.cc:260] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:15.446262 22207 ts_tablet_manager.cc:1428] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260806 04:52:15.457763 22307 ts_tablet_manager.cc:1905] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:15.462858 22210 tablet_bootstrap.cc:492] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36: No bootstrap required, opened a new log
I20260806 04:52:15.463096 22210 ts_tablet_manager.cc:1397] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36: Time spent bootstrapping tablet: real 0.054s user 0.016s sys 0.025s
I20260806 04:52:15.463974 22210 raft_consensus.cc:348] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:15.464264 22210 raft_consensus.cc:374] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:15.464385 22210 raft_consensus.cc:729] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1b1fef3514a09bb73d4a956107a36, State: Initialized, Role: FOLLOWER
I20260806 04:52:15.464740 22210 consensus_queue.cc:260] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:15.467029 22307 ts_tablet_manager.cc:1918] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:52:15.467471 22210 ts_tablet_manager.cc:1428] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260806 04:52:15.477806 22307 log.cc:1196] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/df1e6940eeb3455f80007ef694520f5c
I20260806 04:52:15.478277 22307 ts_tablet_manager.cc:1939] T df1e6940eeb3455f80007ef694520f5c P 53d1b1fef3514a09bb73d4a956107a36: Deleting consensus metadata
I20260806 04:52:15.479249 21674 catalog_manager.cc:4841] TS 53d1b1fef3514a09bb73d4a956107a36 (127.13.148.131:43631): tablet df1e6940eeb3455f80007ef694520f5c (table test-table [id=027d0cc7134d427698db0b41bfc0a88f]) successfully deleted
I20260806 04:52:15.500981 22268 raft_consensus.cc:482] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:52:15.501210 22268 raft_consensus.cc:504] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:15.502066 22268 leader_election.cc:288] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 476a7eb6447240c7a21612c339917d90 (127.13.148.129:40165), 53d1b1fef3514a09bb73d4a956107a36 (127.13.148.131:43631)
I20260806 04:52:15.502918 22131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d479a9e45bdb4c99bcbba0bb13ab5346" candidate_uuid: "46f4d9c56fa54125b72c2256a2e86937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53d1b1fef3514a09bb73d4a956107a36" is_pre_election: true
I20260806 04:52:15.503162 22131 raft_consensus.cc:2454] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46f4d9c56fa54125b72c2256a2e86937 in term 0.
I20260806 04:52:15.504197 21851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d479a9e45bdb4c99bcbba0bb13ab5346" candidate_uuid: "46f4d9c56fa54125b72c2256a2e86937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476a7eb6447240c7a21612c339917d90" is_pre_election: true
I20260806 04:52:15.504492 21851 raft_consensus.cc:2454] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46f4d9c56fa54125b72c2256a2e86937 in term 0.
I20260806 04:52:15.505182 21921 leader_election.cc:302] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46f4d9c56fa54125b72c2256a2e86937, 53d1b1fef3514a09bb73d4a956107a36; no voters:
I20260806 04:52:15.505914 22268 raft_consensus.cc:2781] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:52:15.506047 22268 raft_consensus.cc:482] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:52:15.506140 22268 raft_consensus.cc:3037] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:15.508589 22268 raft_consensus.cc:504] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:15.509548 21851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d479a9e45bdb4c99bcbba0bb13ab5346" candidate_uuid: "46f4d9c56fa54125b72c2256a2e86937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476a7eb6447240c7a21612c339917d90"
I20260806 04:52:15.509754 21851 raft_consensus.cc:3037] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:15.514201 21851 raft_consensus.cc:2454] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46f4d9c56fa54125b72c2256a2e86937 in term 1.
I20260806 04:52:15.514796 21920 leader_election.cc:302] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46f4d9c56fa54125b72c2256a2e86937, 476a7eb6447240c7a21612c339917d90; no voters:
I20260806 04:52:15.515242 22131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d479a9e45bdb4c99bcbba0bb13ab5346" candidate_uuid: "46f4d9c56fa54125b72c2256a2e86937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53d1b1fef3514a09bb73d4a956107a36"
I20260806 04:52:15.515482 22131 raft_consensus.cc:3037] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:15.519186 22131 raft_consensus.cc:2454] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46f4d9c56fa54125b72c2256a2e86937 in term 1.
I20260806 04:52:15.519809 22268 leader_election.cc:288] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [CANDIDATE]: Term 1 election: Requested vote from peers 476a7eb6447240c7a21612c339917d90 (127.13.148.129:40165), 53d1b1fef3514a09bb73d4a956107a36 (127.13.148.131:43631)
I20260806 04:52:15.520071 22268 raft_consensus.cc:2781] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:52:15.520318 22268 raft_consensus.cc:686] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [term 1 LEADER]: Becoming Leader. State: Replica: 46f4d9c56fa54125b72c2256a2e86937, State: Running, Role: LEADER
I20260806 04:52:15.520687 22268 consensus_queue.cc:237] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } }
I20260806 04:52:15.530558 21687 catalog_manager.cc:5495] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46f4d9c56fa54125b72c2256a2e86937 (127.13.148.130). New cstate: current_term: 1 leader_uuid: "46f4d9c56fa54125b72c2256a2e86937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46f4d9c56fa54125b72c2256a2e86937" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 } health_report { overall_health: UNKNOWN } } }
I20260806 04:52:15.565600 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:15.557571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33000 (local address 127.13.148.130:42645)
0806 04:52:15.557763 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:15.557775 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:15.557798 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:52:15.557869 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:15.557874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:15.558043 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:15.558172 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:52:15.558844 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:15.560125 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:15.562584 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:15.563518 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:15.563644 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:15.564282 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:52:15.564471 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:52:15.564557 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:15.565094 (+ 537us) server_negotiation.cc:299] Negotiation successful
0806 04:52:15.565196 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:52:15.594755 22131 raft_consensus.cc:1261] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36 [term 1 FOLLOWER]: Refusing update from remote peer 46f4d9c56fa54125b72c2256a2e86937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:52:15.595758 21851 raft_consensus.cc:1261] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90 [term 1 FOLLOWER]: Refusing update from remote peer 46f4d9c56fa54125b72c2256a2e86937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:52:15.596367 22268 consensus_queue.cc:1035] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1b1fef3514a09bb73d4a956107a36" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:52:15.597383 22268 consensus_queue.cc:1035] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476a7eb6447240c7a21612c339917d90" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:52:15.614001 22300 mvcc.cc:205] Tried to move back new op lower bound from 7315422968186368000 to 7315422967896174592. Current Snapshot: MvccSnapshot[applied={T|T < 7315422968186368000}]
I20260806 04:52:15.689472 21687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:52:15.689755 21687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:52:15.693357 21687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63d61c9de3024b839c4d051e0679a7a6: Sending DeleteTablet for 3 replicas of tablet d479a9e45bdb4c99bcbba0bb13ab5346
I20260806 04:52:15.694038 21831 tablet_service.cc:1514] Processing DeleteTablet for tablet d479a9e45bdb4c99bcbba0bb13ab5346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:15 UTC) from {username='oryx', principal='oryx/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:34644
I20260806 04:52:15.694330 22272 tablet_replica.cc:330] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90: stopping tablet replica
I20260806 04:52:15.694540 22272 raft_consensus.cc:2229] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:52:15.694746 22272 raft_consensus.cc:2258] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:15.695078 21970 tablet_service.cc:1514] Processing DeleteTablet for tablet d479a9e45bdb4c99bcbba0bb13ab5346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:15 UTC) from {username='oryx', principal='oryx/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:32958
I20260806 04:52:15.695371 22274 tablet_replica.cc:330] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937: stopping tablet replica
I20260806 04:52:15.695578 22274 raft_consensus.cc:2229] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:52:15.695923 22274 raft_consensus.cc:2258] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:15.697145 22272 ts_tablet_manager.cc:1905] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:15.701504 22110 tablet_service.cc:1514] Processing DeleteTablet for tablet d479a9e45bdb4c99bcbba0bb13ab5346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:15 UTC) from {username='oryx', principal='oryx/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:49942
I20260806 04:52:15.701669 22274 ts_tablet_manager.cc:1905] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:15.701807 22307 tablet_replica.cc:330] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36: stopping tablet replica
I20260806 04:52:15.702024 22307 raft_consensus.cc:2229] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:52:15.702240 22307 raft_consensus.cc:2258] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:15.703616 22307 ts_tablet_manager.cc:1905] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:15.707695 22274 ts_tablet_manager.cc:1918] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:52:15.707803 22274 log.cc:1196] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/d479a9e45bdb4c99bcbba0bb13ab5346
I20260806 04:52:15.708179 22274 ts_tablet_manager.cc:1939] T d479a9e45bdb4c99bcbba0bb13ab5346 P 46f4d9c56fa54125b72c2256a2e86937: Deleting consensus metadata
I20260806 04:52:15.709116 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 21761
I20260806 04:52:15.712306 22272 ts_tablet_manager.cc:1918] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:52:15.712527 22272 log.cc:1196] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/d479a9e45bdb4c99bcbba0bb13ab5346
I20260806 04:52:15.712960 22272 ts_tablet_manager.cc:1939] T d479a9e45bdb4c99bcbba0bb13ab5346 P 476a7eb6447240c7a21612c339917d90: Deleting consensus metadata
I20260806 04:52:15.713182 22307 ts_tablet_manager.cc:1918] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:52:15.713295 21673 catalog_manager.cc:4841] TS 46f4d9c56fa54125b72c2256a2e86937 (127.13.148.130:42645): tablet d479a9e45bdb4c99bcbba0bb13ab5346 (table test-table [id=a74df609e7054e07883ccae8a2013972]) successfully deleted
I20260806 04:52:15.713620 22307 log.cc:1196] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/d479a9e45bdb4c99bcbba0bb13ab5346
I20260806 04:52:15.714102 22307 ts_tablet_manager.cc:1939] T d479a9e45bdb4c99bcbba0bb13ab5346 P 53d1b1fef3514a09bb73d4a956107a36: Deleting consensus metadata
I20260806 04:52:15.714582 21673 catalog_manager.cc:4841] TS 476a7eb6447240c7a21612c339917d90 (127.13.148.129:40165): tablet d479a9e45bdb4c99bcbba0bb13ab5346 (table test-table [id=a74df609e7054e07883ccae8a2013972]) successfully deleted
I20260806 04:52:15.715371 21674 catalog_manager.cc:4841] TS 53d1b1fef3514a09bb73d4a956107a36 (127.13.148.131:43631): tablet d479a9e45bdb4c99bcbba0bb13ab5346 (table test-table [id=a74df609e7054e07883ccae8a2013972]) successfully deleted
Aug 06 04:52:15 dist-test-slave-1x4g krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.129@KRBTEST.COM
I20260806 04:52:15.735488 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:15.710303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34656 (local address 127.13.148.129:40165)
0806 04:52:15.710498 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:15.710504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:15.722218 (+ 11714us) server_negotiation.cc:407] Connection header received
0806 04:52:15.722416 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:15.722421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:15.722486 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:15.722584 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:15.723709 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:15.724327 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:15.725773 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:15.726169 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:15.726304 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:15.730253 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:15.730266 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:15.730279 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:15.730311 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:15.733085 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:15.733687 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:15.733693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:15.733695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:15.733811 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:15.734165 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:15.734168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:15.734170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:15.734505 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:15.734796 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:15.735025 (+ 229us) server_negotiation.cc:299] Negotiation successful
0806 04:52:15.735217 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 04:52:15.736032 21893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:15.863164 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 21761
I20260806 04:52:15.891813 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 21902
Aug 06 04:52:15 dist-test-slave-1x4g krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.130@KRBTEST.COM
I20260806 04:52:15.906847 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:15.892897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33010 (local address 127.13.148.130:42645)
0806 04:52:15.893068 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:15.893073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:15.893090 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:52:15.893152 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:15.893158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:15.893248 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:15.893345 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:15.894411 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:15.895326 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:15.896345 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:15.897059 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:15.897199 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:15.901154 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:15.901174 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:15.901194 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:15.901234 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:15.904449 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:15.905083 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:15.905091 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:15.905094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:15.905242 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:15.905557 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:15.905561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:15.905564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:15.905980 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:15.906234 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:15.906372 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 04:52:15.906564 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:52:15.907514 22034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:16.039942 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 21902
I20260806 04:52:16.066188 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 22044
Aug 06 04:52:16 dist-test-slave-1x4g krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.131@KRBTEST.COM
I20260806 04:52:16.079984 22322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:16.066887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.13.148.131:43631)
0806 04:52:16.067311 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:16.067315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:16.067330 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:52:16.067384 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:16.067387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:16.067447 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:16.067530 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:16.068835 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:16.069451 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:16.070452 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:16.070811 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:16.070896 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:16.075072 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:16.075086 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:16.075102 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:16.075134 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:16.077742 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:16.078358 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:16.078364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:16.078367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:16.078488 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:16.078755 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:16.078757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:16.078759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:16.079064 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:16.079296 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:16.079523 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 04:52:16.079676 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":226,"threads_started":1}
I20260806 04:52:16.080543 22176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:16.196771 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 22044
I20260806 04:52:16.219722 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 21656
I20260806 04:52:16.220963 21729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:16.326287 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 21656
2026-08-06T04:52:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5644 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:52:16 dist-test-slave-1x4g krb5kdc[22327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:52:16 dist-test-slave-1x4g krb5kdc[22327](info): set up 2 sockets
Aug 06 04:52:16 dist-test-slave-1x4g krb5kdc[22327](info): commencing operation
krb5kdc: starting...
W20260806 04:52:17.365876 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:52:17 dist-test-slave-1x4g krb5kdc[22327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:52:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:52:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:52:17.565965 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41889
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:42619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:41889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:17.863862 22340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:17.864109 22340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:17.864204 22340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:17.870316 22340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:52:17.870406 22340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:17.870453 22340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:17.870508 22340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:52:17.870551 22340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:52:17.879251 22340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:42619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:41889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:17.880311 22340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:17.881490 22340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:17.892067 22348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:17.892133 22345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:17.892098 22346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:17.892693 22340 server_base.cc:1034] running on GCE node
I20260806 04:52:17.893404 22340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:17.894577 22340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:17.895762 22340 hybrid_clock.cc:648] HybridClock initialized: now 1785991937895699 us; error 53 us; skew 500 ppm
I20260806 04:52:17.896023 22340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T04:52:17Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1631 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:52:18 dist-test-slave-1x4g krb5kdc[22354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:52:18 dist-test-slave-1x4g krb5kdc[22354](info): set up 2 sockets
Aug 06 04:52:18 dist-test-slave-1x4g krb5kdc[22354](info): commencing operation
krb5kdc: starting...
W20260806 04:52:18.970490 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:52:19 dist-test-slave-1x4g krb5kdc[22354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:52:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:52:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:52:19.169646 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:42603
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:42603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:19.463897 22367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:19.464164 22367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:19.464300 22367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:19.470690 22367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:52:19.470783 22367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:19.470822 22367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:19.470863 22367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:52:19.470894 22367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:52:19.479393 22367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:42603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:42603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:19.480504 22367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:19.481698 22367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:19.492852 22372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:19.492846 22375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:19.492871 22373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:19.493954 22367 server_base.cc:1034] running on GCE node
I20260806 04:52:19.494726 22367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:19.495939 22367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:19.497162 22367 hybrid_clock.cc:648] HybridClock initialized: now 1785991939497124 us; error 43 us; skew 500 ppm
I20260806 04:52:19.497490 22367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:19 dist-test-slave-1x4g krb5kdc[22354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991939, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:19.500371 22367 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:52:19.501818 22367 webserver.cc:460] Webserver started at http://127.13.148.190:41049/ using document root <none> and password file <none>
I20260806 04:52:19.502445 22367 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:19.502547 22367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:19.502804 22367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:19.505517 22367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "357f40f57b2144a5b277a0d894285c68"
format_stamp: "Formatted at 2026-08-06 04:52:19 on dist-test-slave-1x4g"
server_key: "7128623186d3dac88f0986b7694af5c0"
server_key_iv: "237dfee635927761faf607cf5f0c53f5"
server_key_version: "encryptionkey@0"
I20260806 04:52:19.506152 22367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "357f40f57b2144a5b277a0d894285c68"
format_stamp: "Formatted at 2026-08-06 04:52:19 on dist-test-slave-1x4g"
server_key: "7128623186d3dac88f0986b7694af5c0"
server_key_iv: "237dfee635927761faf607cf5f0c53f5"
server_key_version: "encryptionkey@0"
I20260806 04:52:19.511026 22367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 04:52:19.514106 22382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:19.515239 22367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:52:19.515353 22367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "357f40f57b2144a5b277a0d894285c68"
format_stamp: "Formatted at 2026-08-06 04:52:19 on dist-test-slave-1x4g"
server_key: "7128623186d3dac88f0986b7694af5c0"
server_key_iv: "237dfee635927761faf607cf5f0c53f5"
server_key_version: "encryptionkey@0"
I20260806 04:52:19.515486 22367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:19.533233 22367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:19.536628 22367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:19.536885 22367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:19.553189 22367 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:42603
I20260806 04:52:19.553208 22437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:42603 every 8 connection(s)
I20260806 04:52:19.554278 22367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:52:19.555781 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 22367
I20260806 04:52:19.555980 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:52:19.556527 13906 external_mini_cluster.cc:1428] Setting key 5b02481bacf9f0e2a523ac9d4360dfea
I20260806 04:52:19.557556 22438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:19.567593 22438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68: Bootstrap starting.
I20260806 04:52:19.570138 22438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:19.571094 22438 log.cc:824] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:52:19 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:52:19.573755 22438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68: No bootstrap required, opened a new log
I20260806 04:52:19.578934 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:19.558386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.13.148.190:42603)
0806 04:52:19.559046 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:19.559056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:19.559088 (+ 32us) server_negotiation.cc:407] Connection header received
0806 04:52:19.560435 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:19.560454 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:19.560770 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:19.561191 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:19.562673 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:19.563767 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:19.564858 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:19.565436 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:19.565618 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:19.572620 (+ 7002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:19.572627 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:19.572638 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:19.572667 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:19.575371 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:19.575968 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:19.575974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:19.575979 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:19.576127 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:19.576476 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:19.576482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:19.576485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:19.577043 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:19.577388 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:19.577764 (+ 376us) server_negotiation.cc:299] Negotiation successful
0806 04:52:19.578140 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":297,"threads_started":1}
I20260806 04:52:19.580471 22438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357f40f57b2144a5b277a0d894285c68" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42603 } }
I20260806 04:52:19.580838 22438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:19.580947 22438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357f40f57b2144a5b277a0d894285c68, State: Initialized, Role: FOLLOWER
I20260806 04:52:19.581534 22438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357f40f57b2144a5b277a0d894285c68" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42603 } }
I20260806 04:52:19.581789 22438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:52:19.581929 22438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:52:19.582108 22438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:19.584100 22438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357f40f57b2144a5b277a0d894285c68" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42603 } }
I20260806 04:52:19.584523 22438 leader_election.cc:302] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 357f40f57b2144a5b277a0d894285c68; no voters:
I20260806 04:52:19.585084 22438 leader_election.cc:288] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:52:19.585206 22443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:52:19.585850 22443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [term 1 LEADER]: Becoming Leader. State: Replica: 357f40f57b2144a5b277a0d894285c68, State: Running, Role: LEADER
I20260806 04:52:19.586356 22443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357f40f57b2144a5b277a0d894285c68" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42603 } }
I20260806 04:52:19.587044 22438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:52:19.589094 22444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "357f40f57b2144a5b277a0d894285c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357f40f57b2144a5b277a0d894285c68" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42603 } } }
I20260806 04:52:19.589452 22444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:19.589500 22445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 357f40f57b2144a5b277a0d894285c68. Latest consensus state: current_term: 1 leader_uuid: "357f40f57b2144a5b277a0d894285c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357f40f57b2144a5b277a0d894285c68" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42603 } } }
I20260806 04:52:19.589771 22445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:19.591331 22450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:52:19.595288 22450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:52:19.603729 22450 catalog_manager.cc:1347] Generated new cluster ID: fd1ae5a8110540bf851ec963d0ed6084
I20260806 04:52:19.603821 22450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:52:19.617336 22450 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:52:19.619082 22450 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:52:19.627110 22450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68: Generated new TSK 0
I20260806 04:52:19.627959 22450 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:52:19.675398 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42603
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:19.972275 22464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:19.972615 22464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:19.972769 22464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:19.979292 22464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:19.979432 22464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:19.979621 22464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:52:19.987828 22464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:19.988910 22464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:19.990025 22464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:20.001847 22470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:20.001919 22472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:20.001945 22469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:20.002732 22464 server_base.cc:1034] running on GCE node
I20260806 04:52:20.003301 22464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:20.004123 22464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:20.005333 22464 hybrid_clock.cc:648] HybridClock initialized: now 1785991940005297 us; error 51 us; skew 500 ppm
I20260806 04:52:20.005659 22464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:20 dist-test-slave-1x4g krb5kdc[22354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991940, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:20.008824 22464 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:52:20.010313 22464 webserver.cc:460] Webserver started at http://127.13.148.129:38995/ using document root <none> and password file <none>
I20260806 04:52:20.011024 22464 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:20.011166 22464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:20.011485 22464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:20.014442 22464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "c3cae38bfa024667bb1b0c8daacb1f1c"
format_stamp: "Formatted at 2026-08-06 04:52:20 on dist-test-slave-1x4g"
server_key: "91d0ac8be4773aea0ab4a64c7aefb74b"
server_key_iv: "ee7a31c2d93d63c65218f60cbfc82601"
server_key_version: "encryptionkey@0"
I20260806 04:52:20.015256 22464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "c3cae38bfa024667bb1b0c8daacb1f1c"
format_stamp: "Formatted at 2026-08-06 04:52:20 on dist-test-slave-1x4g"
server_key: "91d0ac8be4773aea0ab4a64c7aefb74b"
server_key_iv: "ee7a31c2d93d63c65218f60cbfc82601"
server_key_version: "encryptionkey@0"
I20260806 04:52:20.020435 22464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 04:52:20.023747 22479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:20.025049 22464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:52:20.025230 22464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "c3cae38bfa024667bb1b0c8daacb1f1c"
format_stamp: "Formatted at 2026-08-06 04:52:20 on dist-test-slave-1x4g"
server_key: "91d0ac8be4773aea0ab4a64c7aefb74b"
server_key_iv: "ee7a31c2d93d63c65218f60cbfc82601"
server_key_version: "encryptionkey@0"
I20260806 04:52:20.025422 22464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:20.039920 22464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:20.043741 22464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:20.044025 22464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:20.044992 22464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:20.046299 22464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:20.046404 22464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:20.046561 22464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:20.046640 22464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:20.079645 22464 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:37501
I20260806 04:52:20.079697 22594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:37501 every 8 connection(s)
I20260806 04:52:20.081182 22464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:52:20.082368 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 22464
I20260806 04:52:20.082599 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:52:20.083400 13906 external_mini_cluster.cc:1428] Setting key bbfa86a1ce5d10c0209e8c6650c59d61
Aug 06 04:52:20 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991940, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:52:20.105177 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:20.084154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:40513 (local address 127.13.148.190:42603)
0806 04:52:20.085206 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:20.085215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:20.085834 (+ 619us) server_negotiation.cc:407] Connection header received
0806 04:52:20.088689 (+ 2855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:20.088693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:20.088751 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:20.088883 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:20.090797 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:20.091480 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:20.094053 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:20.094469 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:20.094561 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:20.098270 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:20.098277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:20.098284 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:20.098324 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:20.102016 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:20.102772 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:20.102779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:20.102784 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:20.102928 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:20.103314 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:20.103321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:20.103326 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:20.103711 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:20.103875 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:20.104492 (+ 617us) server_negotiation.cc:299] Negotiation successful
0806 04:52:20.104725 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":769,"threads_started":1}
I20260806 04:52:20.106032 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:20.084716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:40513)
0806 04:52:20.085663 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:20.085715 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:20.088399 (+ 2684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:20.089149 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:20.089162 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:20.089863 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:20.090628 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:20.090656 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:20.091635 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:20.091643 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:20.093906 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:20.093924 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:20.094691 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:20.094700 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:20.095037 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:20.095848 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:20.095869 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:20.098075 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:20.102149 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:20.102155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:20.102167 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:20.102649 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:20.103029 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:20.103032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:20.103034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:20.103195 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:20.103828 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:20.103833 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:20.104123 (+ 290us) client_negotiation.cc:769] Sending connection context
0806 04:52:20.104318 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 04:52:20.104697 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":361,"threads_started":1}
I20260806 04:52:20.107697 22595 heartbeater.cc:347] Connected to a master server at 127.13.148.190:42603
I20260806 04:52:20.108006 22595 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:20.108728 22595 heartbeater.cc:511] Master 127.13.148.190:42603 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
I20260806 04:52:20.110764 22400 ts_manager.cc:194] Registered new tserver with Master: c3cae38bfa024667bb1b0c8daacb1f1c (127.13.148.129:37501)
I20260806 04:52:20.112262 22400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:40513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:52:20.139472 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42603
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:20.438788 22601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:20.439051 22601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:20.439145 22601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:20.445518 22601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:20.445601 22601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:20.445768 22601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:52:20.454233 22601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:20.455274 22601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:20.456617 22601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:20.469234 22607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:20.469518 22606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:20.469449 22610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:20.470312 22601 server_base.cc:1034] running on GCE node
I20260806 04:52:20.470834 22601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:20.471764 22601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:20.472995 22601 hybrid_clock.cc:648] HybridClock initialized: now 1785991940472944 us; error 58 us; skew 500 ppm
I20260806 04:52:20.473273 22601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:20 dist-test-slave-1x4g krb5kdc[22354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991940, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:20.476054 22601 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:52:20.477413 22601 webserver.cc:460] Webserver started at http://127.13.148.130:37921/ using document root <none> and password file <none>
I20260806 04:52:20.478037 22601 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:20.478142 22601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:20.478400 22601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:20.480918 22601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "6e5de769267d46bfb486c9d0630c9830"
format_stamp: "Formatted at 2026-08-06 04:52:20 on dist-test-slave-1x4g"
server_key: "9a4807ec0b0ba31a067db085eb1147a0"
server_key_iv: "49327a9b35965d505f7d115a4ac89479"
server_key_version: "encryptionkey@0"
I20260806 04:52:20.481624 22601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "6e5de769267d46bfb486c9d0630c9830"
format_stamp: "Formatted at 2026-08-06 04:52:20 on dist-test-slave-1x4g"
server_key: "9a4807ec0b0ba31a067db085eb1147a0"
server_key_iv: "49327a9b35965d505f7d115a4ac89479"
server_key_version: "encryptionkey@0"
I20260806 04:52:20.486150 22601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:52:20.489557 22617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:20.490713 22601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:52:20.490880 22601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "6e5de769267d46bfb486c9d0630c9830"
format_stamp: "Formatted at 2026-08-06 04:52:20 on dist-test-slave-1x4g"
server_key: "9a4807ec0b0ba31a067db085eb1147a0"
server_key_iv: "49327a9b35965d505f7d115a4ac89479"
server_key_version: "encryptionkey@0"
I20260806 04:52:20.491103 22601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:20.515726 22601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:20.519647 22601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:20.519942 22601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:20.520815 22601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:20.522053 22601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:20.522158 22601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:20.522308 22601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:20.522380 22601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:20.553056 22601 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:40231
I20260806 04:52:20.553136 22731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:40231 every 8 connection(s)
I20260806 04:52:20.554214 22601 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:52:20.557603 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 22601
I20260806 04:52:20.557807 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:52:20.558389 13906 external_mini_cluster.cc:1428] Setting key b0622dc6212189302c579aafc13b6d8a
Aug 06 04:52:20 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991940, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:52:20.576417 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:20.556936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:60027 (local address 127.13.148.190:42603)
0806 04:52:20.557141 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:20.557147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:20.558576 (+ 1429us) server_negotiation.cc:407] Connection header received
0806 04:52:20.560915 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:20.560922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:20.561002 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:20.561098 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:20.562916 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:20.564002 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:20.566093 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:20.566781 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:20.566907 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:20.570816 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:20.570820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:20.570823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:20.570868 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:20.573495 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:20.574396 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:20.574400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:20.574402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:20.574517 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:20.574992 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:20.574995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:20.574997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:20.575213 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:20.575316 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:20.576080 (+ 764us) server_negotiation.cc:299] Negotiation successful
0806 04:52:20.576224 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 04:52:20.577978 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:20.557573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:60027)
0806 04:52:20.558412 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:20.558469 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:20.560630 (+ 2161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:20.561334 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:20.561344 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:20.562031 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:20.562743 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:20.562764 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:20.564190 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:20.564195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:20.565945 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:20.565964 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:20.566991 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:20.566997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:20.567249 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:20.567978 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:20.568013 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:20.570632 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:20.573642 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:20.573653 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:20.573669 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:20.574284 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:20.574625 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:20.574633 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:20.574650 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:20.574893 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:20.575321 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:20.575330 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:20.575721 (+ 391us) client_negotiation.cc:769] Sending connection context
0806 04:52:20.576019 (+ 298us) client_negotiation.cc:240] Negotiation successful
0806 04:52:20.576523 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":358,"threads_started":1}
I20260806 04:52:20.579348 22732 heartbeater.cc:347] Connected to a master server at 127.13.148.190:42603
I20260806 04:52:20.579668 22732 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:20.580287 22732 heartbeater.cc:511] Master 127.13.148.190:42603 requested a full tablet report, sending...
I20260806 04:52:20.581677 22400 ts_manager.cc:194] Registered new tserver with Master: 6e5de769267d46bfb486c9d0630c9830 (127.13.148.130:40231)
I20260806 04:52:20.582479 22400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:60027
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
I20260806 04:52:20.611304 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42603
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:20.919749 22738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:20.920027 22738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:20.920185 22738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:20.926647 22738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:20.926769 22738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:20.926946 22738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:52:20.935945 22738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:20.937211 22738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:20.938483 22738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:20.950666 22746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:20.950793 22743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:20.950786 22744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:20.951992 22738 server_base.cc:1034] running on GCE node
I20260806 04:52:20.952504 22738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:20.953369 22738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:20.954553 22738 hybrid_clock.cc:648] HybridClock initialized: now 1785991940954511 us; error 37 us; skew 500 ppm
I20260806 04:52:20.954809 22738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:20 dist-test-slave-1x4g krb5kdc[22354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991940, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:20.957929 22738 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:52:20.959267 22738 webserver.cc:460] Webserver started at http://127.13.148.131:34465/ using document root <none> and password file <none>
I20260806 04:52:20.959896 22738 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:20.960017 22738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:20.960299 22738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:20.962993 22738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "4ae31b7324444e2bb4f81e843040e2be"
format_stamp: "Formatted at 2026-08-06 04:52:20 on dist-test-slave-1x4g"
server_key: "eb2a07b7046975c3e872589fb7af2dc0"
server_key_iv: "eb07385be8b4b9e55260272ab9865000"
server_key_version: "encryptionkey@0"
I20260806 04:52:20.963675 22738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "4ae31b7324444e2bb4f81e843040e2be"
format_stamp: "Formatted at 2026-08-06 04:52:20 on dist-test-slave-1x4g"
server_key: "eb2a07b7046975c3e872589fb7af2dc0"
server_key_iv: "eb07385be8b4b9e55260272ab9865000"
server_key_version: "encryptionkey@0"
I20260806 04:52:20.968076 22738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260806 04:52:20.971196 22753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:20.972421 22738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:52:20.972599 22738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "4ae31b7324444e2bb4f81e843040e2be"
format_stamp: "Formatted at 2026-08-06 04:52:20 on dist-test-slave-1x4g"
server_key: "eb2a07b7046975c3e872589fb7af2dc0"
server_key_iv: "eb07385be8b4b9e55260272ab9865000"
server_key_version: "encryptionkey@0"
I20260806 04:52:20.972810 22738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:20.994099 22738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:20.997786 22738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:20.998077 22738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:20.998899 22738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:21.000089 22738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:21.000177 22738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:21.000259 22738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:21.000303 22738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:21.032265 22738 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:33797
I20260806 04:52:21.032298 22866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:33797 every 8 connection(s)
I20260806 04:52:21.033669 22738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:52:21.040468 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 22738
I20260806 04:52:21.040652 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:52:21.041198 13906 external_mini_cluster.cc:1428] Setting key c1002d9d2e435fe9c25872b59d8507ea
Aug 06 04:52:21 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991940, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:52:21.052896 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:21.036031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:44077 (local address 127.13.148.190:42603)
0806 04:52:21.036244 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:21.036250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:21.037637 (+ 1387us) server_negotiation.cc:407] Connection header received
0806 04:52:21.039394 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:21.039399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:21.039469 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:21.039567 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:21.041231 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:21.042042 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:21.044019 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:21.044493 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:21.044588 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:21.047545 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:21.047550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:21.047553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:21.047583 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:21.050253 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:21.050928 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:21.050932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:21.050934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:21.051067 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:21.051440 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:21.051443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:21.051445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:21.051713 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:21.051830 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:21.052409 (+ 579us) server_negotiation.cc:299] Negotiation successful
0806 04:52:21.052581 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 04:52:21.053642 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:21.036676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:44077)
0806 04:52:21.037472 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:21.037511 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:21.039134 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:21.039724 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:21.039731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:21.040289 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:21.041054 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:21.041078 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:21.042219 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:21.042223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:21.043847 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:21.043863 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:21.044669 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:21.044673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:21.044899 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:21.045501 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:21.045519 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:21.047357 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:21.050375 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:21.050380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:21.050389 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:21.050812 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:21.051159 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:21.051161 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:21.051163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:21.051320 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:21.051816 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:21.051822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:21.052094 (+ 272us) client_negotiation.cc:769] Sending connection context
0806 04:52:21.052295 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 04:52:21.052653 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":291,"threads_started":1}
I20260806 04:52:21.054901 22867 heartbeater.cc:347] Connected to a master server at 127.13.148.190:42603
I20260806 04:52:21.055199 22867 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:21.056015 22867 heartbeater.cc:511] Master 127.13.148.190:42603 requested a full tablet report, sending...
I20260806 04:52:21.057534 22400 ts_manager.cc:194] Registered new tserver with Master: 4ae31b7324444e2bb4f81e843040e2be (127.13.148.131:33797)
I20260806 04:52:21.058302 22400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:44077
I20260806 04:52:21.068337 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.129.keytab.
Entry for principal oryx/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.129.keytab.
I20260806 04:52:21.115242 22595 heartbeater.cc:503] Master 127.13.148.190:42603 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.13.148.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.130.keytab.
Entry for principal oryx/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.130.keytab.
WARNING: no policy specified for oryx/127.13.148.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.131.keytab.
Entry for principal oryx/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.131.keytab.
I20260806 04:52:21.191715 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 22464
Aug 06 04:52:21 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:52:21.206911 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:21.192879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.13.148.129:37501)
0806 04:52:21.193524 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:21.193531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:21.193560 (+ 29us) server_negotiation.cc:407] Connection header received
0806 04:52:21.193745 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:21.193763 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:21.193975 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:21.194140 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:21.195436 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:21.196458 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:21.197684 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:21.198223 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:21.198314 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:21.202056 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:21.202067 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:21.202081 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:21.202118 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:21.204651 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:21.205270 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:21.205276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:21.205278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:21.205392 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:21.205692 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:21.205695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:21.205696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:21.205981 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:21.206198 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:21.206387 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 04:52:21.206555 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":318,"threads_started":1}
I20260806 04:52:21.208072 22590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:21.284432 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 22464
I20260806 04:52:21.302000 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 22601
Aug 06 04:52:21 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:52:21.316169 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:21.302842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.13.148.130:40231)
0806 04:52:21.303417 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:21.303424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:21.303454 (+ 30us) server_negotiation.cc:407] Connection header received
0806 04:52:21.303511 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:21.303516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:21.303717 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:21.303836 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:21.304977 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:21.305907 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:21.307067 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:21.307537 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:21.307625 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:21.311381 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:21.311391 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:21.311422 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:21.311478 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:21.314004 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:21.314562 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:21.314568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:21.314570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:21.314676 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:21.314953 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:21.314956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:21.314957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:21.315241 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:21.315447 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:21.315665 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 04:52:21.315848 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":289,"threads_started":1}
I20260806 04:52:21.317443 22727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:21.391721 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 22601
I20260806 04:52:21.409186 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 22738
Aug 06 04:52:21 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
I20260806 04:52:21.425129 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:21.410198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37904 (local address 127.13.148.131:33797)
0806 04:52:21.410802 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:21.410809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:21.410835 (+ 26us) server_negotiation.cc:407] Connection header received
0806 04:52:21.410891 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:21.410897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:21.411098 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:21.411252 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:21.412617 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:21.413631 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:21.414810 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:21.415338 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:21.415483 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:21.419883 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:21.419893 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:21.419905 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:21.419935 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:21.422542 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:21.423187 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:21.423194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:21.423197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:21.423355 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:21.423701 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:21.423706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:21.423709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:21.424021 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:21.424334 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:21.424548 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 04:52:21.424806 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":313,"threads_started":1}
I20260806 04:52:21.426316 22862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:21.502166 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 22738
I20260806 04:52:21.520925 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 22367
I20260806 04:52:21.522095 22432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:21.626250 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 22367
I20260806 04:52:21.648999 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:42603
--webserver_interface=127.13.148.190
--webserver_port=41049
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:42603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:21.950937 22885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:21.951205 22885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:21.951342 22885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:21.957623 22885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:52:21.957754 22885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:21.957857 22885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:21.957952 22885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:52:21.958032 22885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:52:21.966246 22885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:42603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:42603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=41049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:21.967278 22885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:21.968400 22885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:21.978953 22893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:21.979033 22890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:21.979216 22891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:21.979837 22885 server_base.cc:1034] running on GCE node
I20260806 04:52:21.980535 22885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:21.981719 22885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:21.982904 22885 hybrid_clock.cc:648] HybridClock initialized: now 1785991941982875 us; error 34 us; skew 500 ppm
I20260806 04:52:21.983207 22885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:21 dist-test-slave-1x4g krb5kdc[22354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991941, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:21.985893 22885 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:52:21.987087 22885 webserver.cc:460] Webserver started at http://127.13.148.190:41049/ using document root <none> and password file <none>
I20260806 04:52:21.987666 22885 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:21.987803 22885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:21.992678 22885 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:52:21.995333 22900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:21.996505 22885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:52:21.996697 22885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "357f40f57b2144a5b277a0d894285c68"
format_stamp: "Formatted at 2026-08-06 04:52:19 on dist-test-slave-1x4g"
server_key: "7128623186d3dac88f0986b7694af5c0"
server_key_iv: "237dfee635927761faf607cf5f0c53f5"
server_key_version: "encryptionkey@0"
I20260806 04:52:21.997671 22885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:22.016021 22885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:22.019594 22885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:22.019841 22885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:22.036979 22885 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:42603
I20260806 04:52:22.037014 22951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:42603 every 8 connection(s)
I20260806 04:52:22.038518 22885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:52:22.039774 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 22885
I20260806 04:52:22.040483 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:37501
--local_ip_for_outbound_sockets=127.13.148.129
--tserver_master_addrs=127.13.148.190:42603
--webserver_port=38995
--webserver_interface=127.13.148.129
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.148.129
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:52:22.045014 22952 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:52:22.047832 22952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68: Bootstrap starting.
I20260806 04:52:22.073483 22952 log.cc:824] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68: Log is configured to *not* fsync() on all Append() calls
I20260806 04:52:22.083606 22952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:52:22.084198 22952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68: Bootstrap complete.
I20260806 04:52:22.096813 22952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357f40f57b2144a5b277a0d894285c68" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42603 } }
I20260806 04:52:22.097972 22952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357f40f57b2144a5b277a0d894285c68, State: Initialized, Role: FOLLOWER
I20260806 04:52:22.098711 22952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357f40f57b2144a5b277a0d894285c68" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42603 } }
I20260806 04:52:22.099009 22952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:52:22.099125 22952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:52:22.099305 22952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:52:22.104761 22952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357f40f57b2144a5b277a0d894285c68" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42603 } }
I20260806 04:52:22.105504 22952 leader_election.cc:302] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 357f40f57b2144a5b277a0d894285c68; no voters:
I20260806 04:52:22.106215 22952 leader_election.cc:288] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 04:52:22.106302 22956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:52:22.107673 22956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [term 2 LEADER]: Becoming Leader. State: Replica: 357f40f57b2144a5b277a0d894285c68, State: Running, Role: LEADER
I20260806 04:52:22.108196 22956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357f40f57b2144a5b277a0d894285c68" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42603 } }
I20260806 04:52:22.108459 22952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:52:22.110707 22958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 357f40f57b2144a5b277a0d894285c68. Latest consensus state: current_term: 2 leader_uuid: "357f40f57b2144a5b277a0d894285c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357f40f57b2144a5b277a0d894285c68" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42603 } } }
I20260806 04:52:22.110707 22957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "357f40f57b2144a5b277a0d894285c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357f40f57b2144a5b277a0d894285c68" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42603 } } }
I20260806 04:52:22.111102 22957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:22.111104 22958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68 [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:22.127197 22962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:52:22.132408 22962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:52:22.133746 22962 catalog_manager.cc:1259] Loaded cluster ID: fd1ae5a8110540bf851ec963d0ed6084
I20260806 04:52:22.133880 22962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:52:22.138042 22962 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:52:22.139438 22962 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 357f40f57b2144a5b277a0d894285c68: Loaded TSK: 0
I20260806 04:52:22.140910 22962 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 04:52:22.379357 22954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:22.379591 22954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:22.379712 22954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:22.386081 22954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:22.386163 22954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:22.386338 22954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:52:22.394843 22954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.129.keytab
--principal=oryx/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:37501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=38995
--tserver_master_addrs=127.13.148.190:42603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:22.395949 22954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:22.397220 22954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:22.410683 22980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:22.410933 22979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:22.410663 22982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:22.411875 22954 server_base.cc:1034] running on GCE node
I20260806 04:52:22.412520 22954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:22.413431 22954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:22.414647 22954 hybrid_clock.cc:648] HybridClock initialized: now 1785991942414600 us; error 34 us; skew 500 ppm
I20260806 04:52:22.414917 22954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:22 dist-test-slave-1x4g krb5kdc[22354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991942, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:22.417960 22954 init.cc:377] Logged in from keytab as oryx/127.13.148.129@KRBTEST.COM (short username oryx)
I20260806 04:52:22.419333 22954 webserver.cc:460] Webserver started at http://127.13.148.129:38995/ using document root <none> and password file <none>
I20260806 04:52:22.419906 22954 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:22.420039 22954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:22.424916 22954 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:52:22.427376 22989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:22.428900 22954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:52:22.429055 22954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "c3cae38bfa024667bb1b0c8daacb1f1c"
format_stamp: "Formatted at 2026-08-06 04:52:20 on dist-test-slave-1x4g"
server_key: "91d0ac8be4773aea0ab4a64c7aefb74b"
server_key_iv: "ee7a31c2d93d63c65218f60cbfc82601"
server_key_version: "encryptionkey@0"
I20260806 04:52:22.429937 22954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:22.447466 22954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:22.451176 22954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:22.451428 22954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:22.452334 22954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:22.453620 22954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:22.453735 22954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:22.453853 22954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:22.453931 22954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:22.484812 22954 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:37501
I20260806 04:52:22.484932 23103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:37501 every 8 connection(s)
I20260806 04:52:22.486104 22954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:52:22.496479 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 22954
I20260806 04:52:22.497205 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:40231
--local_ip_for_outbound_sockets=127.13.148.130
--tserver_master_addrs=127.13.148.190:42603
--webserver_port=37921
--webserver_interface=127.13.148.130
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.148.130
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:52:22 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:22 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:22.504583 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:22.488755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:40847)
0806 04:52:22.489629 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:22.489674 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:22.491483 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:22.492929 (+ 1446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:22.492938 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:22.493490 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:22.493967 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:22.493983 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.495592 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.495596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:22.497330 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:22.497349 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.498831 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.498836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:22.499040 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:22.499688 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:22.499714 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:22.503393 (+ 3679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":360,"threads_started":1}
W20260806 04:52:22.504732 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:22.488764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:40847 (local address 127.13.148.190:42603)
0806 04:52:22.490544 (+ 1780us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:22.490559 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:22.490615 (+ 56us) server_negotiation.cc:407] Connection header received
0806 04:52:22.492052 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:22.492065 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:22.492375 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:22.492754 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:22.494203 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.495408 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.497517 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.498593 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.498772 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:22.503739 (+ 4967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:40847: BlockingRecv error: recv got EOF from 127.13.148.129:40847 (error 108)
Metrics: {"server-negotiator.queue_time_us":1450,"thread_start_us":1226,"threads_started":1}
W20260806 04:52:22.505515 23104 heartbeater.cc:644] Failed to heartbeat to 127.13.148.190:42603 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.148.190:42603: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Aug 06 04:52:22 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:22 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:22.513562 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:22.506089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:33997)
0806 04:52:22.506239 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:22.506256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:22.506362 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:22.506730 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:22.506734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:22.507072 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:22.507191 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:22.507198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.508108 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.508115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:22.509183 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:22.509195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.509869 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.509874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:22.510061 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:22.510629 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:22.510644 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:22.513309 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 04:52:22.514007 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:22.506272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:33997 (local address 127.13.148.190:42603)
0806 04:52:22.506473 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:22.506477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:22.506489 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:52:22.506526 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:22.506529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:22.506596 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:22.506705 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:22.507319 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.507976 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.509330 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.509731 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.509874 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:22.513503 (+ 3629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:33997: BlockingRecv error: recv got EOF from 127.13.148.129:33997 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 04:52:22 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:22 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:22.522142 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:22.514218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:54375)
0806 04:52:22.514370 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:22.514384 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:22.514490 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:22.515061 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:22.515066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:22.515332 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:22.515458 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:22.515465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.516976 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.516979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:22.517872 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:22.517881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.518836 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.518840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:22.518976 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:22.519499 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:22.519513 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:22.521953 (+ 2440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 04:52:22.522508 23104 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 04:52:22.522910 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:22.514547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:54375 (local address 127.13.148.190:42603)
0806 04:52:22.514736 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:22.514743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:22.514764 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:52:22.514813 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:22.514821 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:22.514909 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:22.515013 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:22.515590 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.516830 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.518007 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.518708 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.518846 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:22.522225 (+ 3379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:54375: BlockingRecv error: recv got EOF from 127.13.148.129:54375 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 04:52:22.813086 23108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:22.813300 23108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:22.813380 23108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:22.819701 23108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:22.819818 23108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:22.819955 23108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:52:22.829963 23108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.130.keytab
--principal=oryx/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:40231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=37921
--tserver_master_addrs=127.13.148.190:42603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:22.831588 23108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:22.833278 23108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:22.848091 23113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:22.848168 23114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:22.848479 23116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:22.849735 23108 server_base.cc:1034] running on GCE node
I20260806 04:52:22.850412 23108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:22.851569 23108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:22.852878 23108 hybrid_clock.cc:648] HybridClock initialized: now 1785991942852786 us; error 83 us; skew 500 ppm
I20260806 04:52:22.853223 23108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:22 dist-test-slave-1x4g krb5kdc[22354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991942, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:22.856097 23108 init.cc:377] Logged in from keytab as oryx/127.13.148.130@KRBTEST.COM (short username oryx)
I20260806 04:52:22.857388 23108 webserver.cc:460] Webserver started at http://127.13.148.130:37921/ using document root <none> and password file <none>
I20260806 04:52:22.858027 23108 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:22.858137 23108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:22.863025 23108 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:52:22.865335 23123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:22.866497 23108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:52:22.866659 23108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "6e5de769267d46bfb486c9d0630c9830"
format_stamp: "Formatted at 2026-08-06 04:52:20 on dist-test-slave-1x4g"
server_key: "9a4807ec0b0ba31a067db085eb1147a0"
server_key_iv: "49327a9b35965d505f7d115a4ac89479"
server_key_version: "encryptionkey@0"
I20260806 04:52:22.867584 23108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:22.886583 23108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:22.890184 23108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:22.890424 23108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:22.891283 23108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:22.892588 23108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:22.892715 23108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:22.892901 23108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:22.892993 23108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:22.923243 23108 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:40231
I20260806 04:52:22.923326 23237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:40231 every 8 connection(s)
I20260806 04:52:22.924468 23108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:52:22.931567 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 23108
I20260806 04:52:22.932205 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:33797
--local_ip_for_outbound_sockets=127.13.148.131
--tserver_master_addrs=127.13.148.190:42603
--webserver_port=34465
--webserver_interface=127.13.148.131
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.148.131
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:52:22 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:22 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 04:52:22.940568 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:22.926813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:53497 (local address 127.13.148.190:42603)
0806 04:52:22.926988 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:22.926993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:22.928022 (+ 1029us) server_negotiation.cc:407] Connection header received
0806 04:52:22.929777 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:22.929781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:22.929839 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:22.929914 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:22.931221 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.931993 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.934390 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.934744 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.934828 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:22.940399 (+ 5571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:53497: BlockingRecv error: recv got EOF from 127.13.148.130:53497 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:52:22.941335 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:22.927215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:53497)
0806 04:52:22.927860 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:22.927899 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:22.929560 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:22.930118 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:22.930125 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:22.930562 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:22.931061 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:22.931077 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.932158 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.932164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:22.934260 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:22.934275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.934948 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.934956 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:22.935224 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:22.935836 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:22.935865 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:22.940269 (+ 4404us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":265,"threads_started":1}
W20260806 04:52:22.941913 23238 heartbeater.cc:644] Failed to heartbeat to 127.13.148.190:42603 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.148.190:42603: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Aug 06 04:52:22 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:22 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:22.949697 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:22.942486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:51145)
0806 04:52:22.942638 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:22.942658 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:22.942772 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:22.943079 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:22.943099 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:22.943345 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:22.943458 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:22.943464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.944725 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.944733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:22.945536 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:22.945545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.946434 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.946437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:22.946544 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:22.947012 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:22.947026 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:22.949540 (+ 2514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260806 04:52:22.950039 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:22.942622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:51145 (local address 127.13.148.190:42603)
0806 04:52:22.942828 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:22.942835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:22.942852 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:52:22.942898 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:22.942904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:22.942971 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:22.943057 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:22.943574 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.944580 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.945665 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.946313 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.946436 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:22.949785 (+ 3349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:51145: BlockingRecv error: recv got EOF from 127.13.148.130:51145 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 04:52:22 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:22 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:22.958264 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:22.950657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:46695)
0806 04:52:22.950803 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:22.950820 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:22.950908 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:22.951232 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:22.951234 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:22.951450 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:22.951591 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:22.951598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.953087 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.953090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:22.953888 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:22.953897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.954844 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.954848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:22.954957 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:22.955468 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:22.955482 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:22.958096 (+ 2614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 04:52:22.958583 23238 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 04:52:22.958956 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:22.950796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:46695 (local address 127.13.148.190:42603)
0806 04:52:22.950980 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:22.950987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:22.951005 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:52:22.951049 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:22.951056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:22.951123 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:22.951211 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:22.951725 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.952968 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.954019 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:22.954722 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:22.954856 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:22.958707 (+ 3851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:46695: BlockingRecv error: recv got EOF from 127.13.148.130:46695 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 04:52:23.235993 23241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:23.236207 23241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:23.236284 23241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:23.243607 23241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:23.243690 23241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:23.243784 23241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:52:23.251801 23241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/krb5kdc/oryx_127.13.148.131.keytab
--principal=oryx/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:33797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=34465
--tserver_master_addrs=127.13.148.190:42603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:23.252871 23241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:23.253875 23241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:23.265839 23246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:23.265848 23247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:23.265777 23249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:23.266896 23241 server_base.cc:1034] running on GCE node
I20260806 04:52:23.267493 23241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:23.268347 23241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:23.269556 23241 hybrid_clock.cc:648] HybridClock initialized: now 1785991943269532 us; error 46 us; skew 500 ppm
I20260806 04:52:23.269860 23241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:23 dist-test-slave-1x4g krb5kdc[22354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991943, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:23.272859 23241 init.cc:377] Logged in from keytab as oryx/127.13.148.131@KRBTEST.COM (short username oryx)
I20260806 04:52:23.274081 23241 webserver.cc:460] Webserver started at http://127.13.148.131:34465/ using document root <none> and password file <none>
I20260806 04:52:23.274653 23241 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:23.274791 23241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:23.279490 23241 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260806 04:52:23.281726 23256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:23.282936 23241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:52:23.283087 23241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "4ae31b7324444e2bb4f81e843040e2be"
format_stamp: "Formatted at 2026-08-06 04:52:20 on dist-test-slave-1x4g"
server_key: "eb2a07b7046975c3e872589fb7af2dc0"
server_key_iv: "eb07385be8b4b9e55260272ab9865000"
server_key_version: "encryptionkey@0"
I20260806 04:52:23.283979 23241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:23.307334 23241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:23.311064 23241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:23.311311 23241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:23.312249 23241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:23.313494 23241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:23.313614 23241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:23.313741 23241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:23.313820 23241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:23.343070 23241 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:33797
I20260806 04:52:23.343144 23370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:33797 every 8 connection(s)
I20260806 04:52:23.344319 23241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:52:23.348284 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 23241
Aug 06 04:52:23 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:23 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 04:52:23.359125 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:23.346647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:46101 (local address 127.13.148.190:42603)
0806 04:52:23.346844 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:23.346848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:23.347925 (+ 1077us) server_negotiation.cc:407] Connection header received
0806 04:52:23.350179 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:23.350184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:23.350275 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:23.350383 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:23.351658 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.352459 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.353917 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.354336 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.354462 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:23.358880 (+ 4418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.131:46101: BlockingRecv error: recv got EOF from 127.13.148.131:46101 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:52:23.359653 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:23.347111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:46101)
0806 04:52:23.347744 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:23.347776 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:23.349953 (+ 2177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:23.350583 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:23.350590 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:23.351013 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:23.351470 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:23.351485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.352617 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.352621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:23.353775 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:23.353786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.354546 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.354553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:23.354743 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:23.355267 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:23.355285 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:23.358709 (+ 3424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":261,"threads_started":1}
W20260806 04:52:23.360137 23371 heartbeater.cc:644] Failed to heartbeat to 127.13.148.190:42603 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.148.190:42603: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
I20260806 04:52:23.362946 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:23.349342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.13.148.190:42603)
0806 04:52:23.349994 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:23.350001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:23.350017 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:52:23.350080 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:23.350084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:23.350135 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:23.350235 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:23.351550 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.352337 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.353596 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.353972 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.354101 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:23.356671 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:23.356679 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:23.356691 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:23.356723 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:23.359672 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:23.360483 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:23.360491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:23.360504 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:23.360679 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:23.361128 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:23.361134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:23.361137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:23.361578 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:23.361974 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:23.362312 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 04:52:23.362660 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":550,"thread_start_us":438,"threads_started":1}
Aug 06 04:52:23 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:23 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:23.367686 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:23.360625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:56399)
0806 04:52:23.360765 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:23.360797 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:23.360875 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:23.361140 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:23.361144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:23.361435 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:23.361553 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:23.361578 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.362574 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.362580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:23.363554 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:23.363565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.364204 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.364209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:23.364360 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:23.364931 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:23.364963 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:23.367455 (+ 2492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 04:52:23.367817 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:23.360696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:56399 (local address 127.13.148.190:42603)
0806 04:52:23.360847 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:23.360850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:23.360862 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:52:23.360976 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:23.360980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:23.361036 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:23.361118 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:23.361719 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.362434 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.363675 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.364074 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.364194 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:23.367619 (+ 3425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.131:56399: BlockingRecv error: recv got EOF from 127.13.148.131:56399 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 04:52:23 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:23 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:23.374773 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:23.368230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:45309)
0806 04:52:23.368341 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:23.368353 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:23.368442 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:23.368731 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:23.368735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:23.369005 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:23.369119 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:23.369126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.369978 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.369986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:23.370978 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:23.370989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.371715 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.371721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:23.371887 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:23.372397 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:23.372413 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:23.374518 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 04:52:23.374897 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:23.368302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:45309 (local address 127.13.148.190:42603)
0806 04:52:23.368533 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:23.368537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:23.368549 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:52:23.368588 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:23.368590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:23.368628 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:23.368710 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:23.369238 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.369833 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.371109 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.371570 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.371711 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:23.374669 (+ 2958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.131:45309: BlockingRecv error: recv got EOF from 127.13.148.131:45309 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
W20260806 04:52:23.375030 23371 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 04:52:23 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:23 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:23.531114 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:23.523236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:49817)
0806 04:52:23.523719 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:23.523735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:23.523843 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:23.524204 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:23.524207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:23.524654 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:23.524891 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:23.524901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.525803 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.525806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:23.526724 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:23.526734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.527353 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.527357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:23.527469 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:23.528074 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:23.528090 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:23.530892 (+ 2802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":280,"threads_started":1}
W20260806 04:52:23.531415 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:23.523292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:49817 (local address 127.13.148.190:42603)
0806 04:52:23.523531 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:23.523538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:23.523816 (+ 278us) server_negotiation.cc:407] Connection header received
0806 04:52:23.523948 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:23.523983 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:23.524059 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:23.524155 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:23.525020 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.525694 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.526831 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.527223 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.527301 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:23.531089 (+ 3788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:49817: BlockingRecv error: recv got EOF from 127.13.148.129:49817 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 04:52:23 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:23 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:23.966885 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:23.959318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:44173)
0806 04:52:23.959853 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:23.959869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:23.960018 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:23.960418 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:23.960423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:23.961017 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:23.961303 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:23.961316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.962277 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.962281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:23.963163 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:23.963172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.963743 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.963746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:23.963851 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:23.964322 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:23.964337 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:23.966696 (+ 2359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":254,"threads_started":1}
W20260806 04:52:23.967074 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:23.959349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:44173 (local address 127.13.148.190:42603)
0806 04:52:23.959566 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:23.959575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:23.959945 (+ 370us) server_negotiation.cc:407] Connection header received
0806 04:52:23.960168 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:23.960172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:23.960248 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:23.960349 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:23.961440 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.962161 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.963267 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:23.963616 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:23.963691 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:23.966869 (+ 3178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:44173: BlockingRecv error: recv got EOF from 127.13.148.130:44173 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 04:52:24 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:24 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:24.383985 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:24.375676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:40071)
0806 04:52:24.376143 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:24.376158 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:24.376302 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:24.376629 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:24.376634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:24.377256 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:24.377453 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:24.377462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:24.378495 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:24.378502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:24.379688 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:24.379700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:24.380438 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:24.380447 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:24.380553 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:24.381174 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:24.381190 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:24.383786 (+ 2596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":251,"threads_started":1}
W20260806 04:52:24.384148 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:24.375761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:40071 (local address 127.13.148.190:42603)
0806 04:52:24.375980 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:24.375988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:24.376237 (+ 249us) server_negotiation.cc:407] Connection header received
0806 04:52:24.376432 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:24.376436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:24.376495 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:24.376589 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:24.377597 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:24.378350 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:24.379837 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:24.380286 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:24.380383 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:24.383963 (+ 3580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.131:40071: BlockingRecv error: recv got EOF from 127.13.148.131:40071 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 04:52:24 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:24 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:24.539577 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:24.532021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:49295)
0806 04:52:24.532511 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:24.532527 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:24.532636 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:24.533021 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:24.533024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:24.533362 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:24.533522 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:24.533529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:24.534476 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:24.534479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:24.535251 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:24.535261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:24.535940 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:24.535944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:24.536064 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:24.536607 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:24.536624 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:24.539358 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":272,"threads_started":1}
W20260806 04:52:24.539745 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:24.532036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:49295 (local address 127.13.148.190:42603)
0806 04:52:24.532283 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:24.532293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:24.532606 (+ 313us) server_negotiation.cc:407] Connection header received
0806 04:52:24.532824 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:24.532830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:24.532906 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:24.532982 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:24.533646 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:24.534365 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:24.535390 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:24.535809 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:24.535902 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:24.539511 (+ 3609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:49295: BlockingRecv error: recv got EOF from 127.13.148.129:49295 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 04:52:24 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:24 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:24.980079 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:24.967902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:60097)
0806 04:52:24.968400 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:24.968414 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:24.968566 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:24.968892 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:24.968896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:24.969295 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:24.969457 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:24.969465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:24.970365 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:24.970372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:24.971248 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:24.971259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:24.971994 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:24.972003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:24.972141 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:24.977169 (+ 5028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:24.977187 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:24.979856 (+ 2669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":282,"threads_started":1}
W20260806 04:52:24.980230 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:24.968108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:60097 (local address 127.13.148.190:42603)
0806 04:52:24.968303 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:24.968309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:24.968501 (+ 192us) server_negotiation.cc:407] Connection header received
0806 04:52:24.968677 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:24.968682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:24.968753 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:24.968859 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:24.969582 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:24.970235 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:24.971375 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:24.971804 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:24.971951 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:24.980042 (+ 8091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:60097: BlockingRecv error: recv got EOF from 127.13.148.130:60097 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 04:52:25 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:25 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:25.392825 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:25.384939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:57289)
0806 04:52:25.385421 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:25.385436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:25.385534 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:25.385784 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:25.385786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:25.386151 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:25.386300 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:25.386308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:25.387222 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:25.387226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:25.388070 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:25.388081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:25.389019 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:25.389024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:25.389135 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:25.389661 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:25.389694 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:25.392546 (+ 2852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":270,"threads_started":1}
W20260806 04:52:25.393049 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:25.385051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:57289 (local address 127.13.148.190:42603)
0806 04:52:25.385196 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:25.385200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:25.385500 (+ 300us) server_negotiation.cc:407] Connection header received
0806 04:52:25.385617 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:25.385620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:25.385671 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:25.385739 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:25.386406 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:25.387110 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:25.388279 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:25.388881 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:25.388985 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:25.392723 (+ 3738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.131:57289: BlockingRecv error: recv got EOF from 127.13.148.131:57289 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 06 04:52:25 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:25 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:25.548027 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:25.540464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:41727)
0806 04:52:25.540931 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:25.540945 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:25.541085 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:25.541449 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:25.541454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:25.541923 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:25.542157 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:25.542170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:25.543223 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:25.543226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:25.544004 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:25.544014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:25.544545 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:25.544548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:25.544654 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:25.545347 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:25.545363 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:25.547775 (+ 2412us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":248,"threads_started":1}
W20260806 04:52:25.548177 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:25.540596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:41727 (local address 127.13.148.190:42603)
0806 04:52:25.540811 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:25.540819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:25.541036 (+ 217us) server_negotiation.cc:407] Connection header received
0806 04:52:25.541217 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:25.541221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:25.541293 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:25.541385 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:25.542318 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:25.543102 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:25.544119 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:25.544439 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:25.544520 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:25.547978 (+ 3458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:41727: BlockingRecv error: recv got EOF from 127.13.148.129:41727 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 06 04:52:25 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:25 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:25.988613 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:25.981002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:42085)
0806 04:52:25.981414 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:25.981428 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:25.981529 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:25.981809 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:25.981812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:25.982141 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:25.982313 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:25.982321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:25.983401 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:25.983405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:25.984387 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:25.984398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:25.985133 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:25.985138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:25.985286 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:25.985758 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:25.985771 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:25.988283 (+ 2512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":224,"threads_started":1}
W20260806 04:52:25.988736 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:25.981118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:42085 (local address 127.13.148.190:42603)
0806 04:52:25.981304 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:25.981309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:25.981507 (+ 198us) server_negotiation.cc:407] Connection header received
0806 04:52:25.981618 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:25.981621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:25.981677 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:25.981748 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:25.982435 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:25.983249 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:25.984510 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:25.985000 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:25.985114 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:25.988500 (+ 3386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:42085: BlockingRecv error: recv got EOF from 127.13.148.130:42085 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 04:52:26 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:26 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:26.401398 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:26.393736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:47495)
0806 04:52:26.394220 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:26.394234 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:26.394342 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:26.394583 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:26.394586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:26.394906 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:26.395069 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:26.395077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:26.395945 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:26.395954 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:26.396874 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:26.396885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:26.397526 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:26.397530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:26.397647 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:26.398286 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:26.398302 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:26.401005 (+ 2703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":251,"threads_started":1}
W20260806 04:52:26.401427 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:26.393778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:47495 (local address 127.13.148.190:42603)
0806 04:52:26.394001 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:26.394010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:26.394311 (+ 301us) server_negotiation.cc:407] Connection header received
0806 04:52:26.394441 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:26.394444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:26.394501 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:26.394571 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:26.395188 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:26.395837 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:26.396996 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:26.397409 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:26.397499 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:26.401213 (+ 3714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.131:47495: BlockingRecv error: recv got EOF from 127.13.148.131:47495 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 06 04:52:26 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:26 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:26.557054 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:26.548937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:35775)
0806 04:52:26.549433 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:26.549447 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:26.549588 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:26.549895 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:26.549898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:26.550271 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:26.550406 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:26.550414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:26.551320 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:26.551325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:26.552296 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:26.552308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:26.553125 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:26.553130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:26.553298 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:26.553834 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:26.553849 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:26.556706 (+ 2857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":276,"threads_started":1}
W20260806 04:52:26.557181 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:26.549132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:35775 (local address 127.13.148.190:42603)
0806 04:52:26.549370 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:26.549376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:26.549518 (+ 142us) server_negotiation.cc:407] Connection header received
0806 04:52:26.549697 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:26.549702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:26.549764 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:26.549855 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:26.550531 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:26.551197 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:26.552466 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:26.552971 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:26.553103 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:26.556925 (+ 3822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:35775: BlockingRecv error: recv got EOF from 127.13.148.129:35775 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 06 04:52:26 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:26 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:26.996934 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:26.989458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:51581)
0806 04:52:26.989876 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:26.989891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:26.990040 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:26.990417 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:26.990420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:26.990788 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:26.990937 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:26.990945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:26.991959 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:26.991962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:26.992922 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:26.992934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:26.993590 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:26.993594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:26.993700 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:26.994219 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:26.994234 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:26.996655 (+ 2421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":232,"threads_started":1}
W20260806 04:52:26.997159 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:26.989492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:51581 (local address 127.13.148.190:42603)
0806 04:52:26.989696 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:26.989705 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:26.990044 (+ 339us) server_negotiation.cc:407] Connection header received
0806 04:52:26.990170 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:26.990175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:26.990267 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:26.990388 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:26.991068 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:26.991811 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:26.993084 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:26.993474 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:26.993567 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:26.996934 (+ 3367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:51581: BlockingRecv error: recv got EOF from 127.13.148.130:51581 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 04:52:27 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:27 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:27.411177 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:27.402280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:59069)
0806 04:52:27.402765 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:27.402779 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:27.402935 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:27.403276 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:27.403280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:27.403687 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:27.403824 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:27.403832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:27.405089 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:27.405127 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:27.406249 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:27.406260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:27.406907 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:27.406912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:27.407082 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:27.407729 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:27.407746 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:27.410966 (+ 3220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":265,"threads_started":1}
W20260806 04:52:27.411391 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:27.402359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:59069 (local address 127.13.148.190:42603)
0806 04:52:27.402596 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:27.402605 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:27.402862 (+ 257us) server_negotiation.cc:407] Connection header received
0806 04:52:27.403086 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:27.403090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:27.403154 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:27.403242 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:27.403960 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:27.404957 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:27.406380 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:27.406772 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:27.406884 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:27.411144 (+ 4260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.131:59069: BlockingRecv error: recv got EOF from 127.13.148.131:59069 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 06 04:52:27 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:27 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:27.565883 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:27.558075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:51127)
0806 04:52:27.558521 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:27.558536 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:27.558676 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:27.559035 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:27.559039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:27.559602 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:27.559828 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:27.559841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:27.560962 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:27.560965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:27.561789 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:27.561798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:27.562447 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:27.562458 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:27.562578 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:27.563200 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:27.563216 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:27.565674 (+ 2458us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":220,"threads_started":1}
W20260806 04:52:27.566032 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:27.558112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:51127 (local address 127.13.148.190:42603)
0806 04:52:27.558339 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:27.558345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:27.558626 (+ 281us) server_negotiation.cc:407] Connection header received
0806 04:52:27.558789 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:27.558792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:27.558876 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:27.558975 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:27.559966 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:27.560846 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:27.561919 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:27.562334 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:27.562420 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:27.565837 (+ 3417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:51127: BlockingRecv error: recv got EOF from 127.13.148.129:51127 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 04:52:27 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:27 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:28.005616 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:27.997832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:54453)
0806 04:52:27.998294 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:27.998309 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:27.998407 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:27.998657 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:27.998660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:27.999047 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:27.999206 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:27.999214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:28.000079 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:28.000082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:28.000962 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:28.000974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:28.001600 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:28.001604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:28.001796 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:28.002534 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:28.002562 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:28.005386 (+ 2824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":245,"threads_started":1}
W20260806 04:52:28.005765 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:27.998040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:54453 (local address 127.13.148.190:42603)
0806 04:52:27.998234 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:27.998241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:27.998375 (+ 134us) server_negotiation.cc:407] Connection header received
0806 04:52:27.998491 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:27.998494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:27.998548 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:27.998616 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:27.999323 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:27.999976 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:28.001100 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:28.001491 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:28.001572 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:28.005553 (+ 3981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:54453: BlockingRecv error: recv got EOF from 127.13.148.130:54453 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 06 04:52:28 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:28 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:28.419591 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:28.412019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:37343)
0806 04:52:28.412554 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:28.412570 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:28.412677 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:28.412955 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:28.412958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:28.413358 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:28.413506 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:28.413514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:28.414340 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:28.414343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:28.415159 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:28.415168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:28.415897 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:28.415900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:28.416008 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:28.416487 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:28.416502 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:28.419273 (+ 2771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":256,"threads_started":1}
W20260806 04:52:28.419749 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:28.412094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:37343 (local address 127.13.148.190:42603)
0806 04:52:28.412266 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:28.412270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:28.412640 (+ 370us) server_negotiation.cc:407] Connection header received
0806 04:52:28.412797 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:28.412801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:28.412860 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:28.412935 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:28.413624 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:28.414234 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:28.415259 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:28.415788 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:28.415871 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:28.419549 (+ 3678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.131:37343: BlockingRecv error: recv got EOF from 127.13.148.131:37343 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 06 04:52:28 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:28 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:28.574749 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:28.566889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:47099)
0806 04:52:28.567441 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:28.567457 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:28.567566 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:28.567848 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:28.567850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:28.568238 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:28.568388 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:28.568396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:28.569329 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:28.569332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:28.570109 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:28.570118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:28.570766 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:28.570770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:28.570874 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:28.571407 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:28.571422 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:28.574478 (+ 3056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":299,"threads_started":1}
W20260806 04:52:28.574898 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:28.566989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:47099 (local address 127.13.148.190:42603)
0806 04:52:28.567185 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:28.567189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:28.567524 (+ 335us) server_negotiation.cc:407] Connection header received
0806 04:52:28.567656 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:28.567660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:28.567738 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:28.567834 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:28.568495 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:28.569218 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:28.570216 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:28.570648 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:28.570734 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:28.574656 (+ 3922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:47099: BlockingRecv error: recv got EOF from 127.13.148.129:47099 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 04:52:29 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:29 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:29.014266 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:29.006638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:40915)
0806 04:52:29.007163 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:29.007177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:29.007297 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:29.007562 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:29.007564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:29.007905 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:29.008096 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:29.008104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:29.009113 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:29.009117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:29.010105 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:29.010116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:29.010690 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:29.010694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:29.010800 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:29.011391 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:29.011407 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:29.014076 (+ 2669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":298,"threads_started":1}
W20260806 04:52:29.014478 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:29.006678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:40915 (local address 127.13.148.190:42603)
0806 04:52:29.006825 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:29.006829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:29.007256 (+ 427us) server_negotiation.cc:407] Connection header received
0806 04:52:29.007382 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:29.007386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:29.007442 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:29.007522 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:29.008224 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:29.008994 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:29.010214 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:29.010559 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:29.010658 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:29.014265 (+ 3607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:40915: BlockingRecv error: recv got EOF from 127.13.148.130:40915 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 04:52:29 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:29 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:29.428200 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:29.420525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:40929)
0806 04:52:29.421036 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:29.421050 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:29.421153 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:29.421414 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:29.421416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:29.421778 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:29.421923 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:29.421930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:29.423430 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:29.423435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:29.424286 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:29.424297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:29.424957 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:29.424960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:29.425063 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:29.425556 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:29.425571 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:29.427933 (+ 2362us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":297,"threads_started":1}
W20260806 04:52:29.428357 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:29.420723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:40929 (local address 127.13.148.190:42603)
0806 04:52:29.420887 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:29.420890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:29.421119 (+ 229us) server_negotiation.cc:407] Connection header received
0806 04:52:29.421250 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:29.421253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:29.421306 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:29.421369 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:29.422082 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:29.423271 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:29.424403 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:29.424829 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:29.424923 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:29.428105 (+ 3182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.131:40929: BlockingRecv error: recv got EOF from 127.13.148.131:40929 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 04:52:29 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:29 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:29.582908 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:29.575606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:42039)
0806 04:52:29.576164 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:29.576179 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:29.576285 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:29.576552 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:29.576556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:29.577037 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:29.577210 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:29.577218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:29.578140 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:29.578143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:29.578929 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:29.578938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:29.579502 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:29.579505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:29.579605 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:29.580128 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:29.580143 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:29.582663 (+ 2520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":279,"threads_started":1}
W20260806 04:52:29.583050 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:29.575691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:42039 (local address 127.13.148.190:42603)
0806 04:52:29.575854 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:29.575858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:29.576248 (+ 390us) server_negotiation.cc:407] Connection header received
0806 04:52:29.576374 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:29.576377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:29.576431 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:29.576509 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:29.577328 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:29.578032 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:29.579042 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:29.579390 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:29.579474 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:29.582859 (+ 3385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:42039: BlockingRecv error: recv got EOF from 127.13.148.129:42039 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 04:52:30 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:30 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:30.029136 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:30.015416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:50059)
0806 04:52:30.016042 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:30.016058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:30.016216 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:30.016576 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:30.016580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:30.017110 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:30.017279 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:30.017288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:30.018362 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:30.018367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:30.019415 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:30.019427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:30.020129 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:30.020134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:30.020302 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:30.025677 (+ 5375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:30.025694 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:30.028891 (+ 3197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":299,"threads_started":1}
W20260806 04:52:30.029261 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:30.015531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:50059 (local address 127.13.148.190:42603)
0806 04:52:30.015713 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:30.015721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:30.016149 (+ 428us) server_negotiation.cc:407] Connection header received
0806 04:52:30.016348 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:30.016354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:30.016433 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:30.016537 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:30.017423 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:30.018218 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:30.019560 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:30.019972 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:30.020099 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:30.029045 (+ 8946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:50059: BlockingRecv error: recv got EOF from 127.13.148.130:50059 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 06 04:52:30 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:30 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:30.437170 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:30.429123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:53747)
0806 04:52:30.429674 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:30.429689 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:30.429789 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:30.430044 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:30.430047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:30.430436 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:30.430640 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:30.430649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:30.431555 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:30.431558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:30.432477 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:30.432487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:30.433485 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:30.433490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:30.433670 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:30.434352 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:30.434370 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:30.436915 (+ 2545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":257,"threads_started":1}
W20260806 04:52:30.437294 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:30.429176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:53747 (local address 127.13.148.190:42603)
0806 04:52:30.429324 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:30.429328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:30.429752 (+ 424us) server_negotiation.cc:407] Connection header received
0806 04:52:30.429873 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:30.429876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:30.429956 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:30.430031 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:30.430757 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:30.431446 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:30.432648 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:30.433312 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:30.433461 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:30.437106 (+ 3645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.131:53747: BlockingRecv error: recv got EOF from 127.13.148.131:53747 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 06 04:52:30 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:30 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:30.592164 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:30.584036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:58809)
0806 04:52:30.584589 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:30.584605 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:30.584729 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:30.585125 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:30.585130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:30.585573 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:30.585781 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:30.585794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:30.586889 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:30.586894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:30.587715 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:30.587724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:30.588564 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:30.588571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:30.588742 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:30.589379 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:30.589403 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:30.591826 (+ 2423us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":300,"threads_started":1}
W20260806 04:52:30.592273 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:30.584339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:58809 (local address 127.13.148.190:42603)
0806 04:52:30.584521 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:30.584525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:30.584701 (+ 176us) server_negotiation.cc:407] Connection header received
0806 04:52:30.584877 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:30.584882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:30.584964 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:30.585084 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:30.585929 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:30.586717 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:30.587880 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:30.588371 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:30.588516 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:30.591999 (+ 3483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:58809: BlockingRecv error: recv got EOF from 127.13.148.129:58809 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 04:52:31 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:31 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:31.038113 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:31.030012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:45631)
0806 04:52:31.030553 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:31.030568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:31.030686 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:31.030948 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:31.030951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:31.031321 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:31.031468 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:31.031475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:31.032583 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:31.032607 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:31.033836 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:31.033848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:31.034519 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:31.034523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:31.034627 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:31.035136 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:31.035152 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:31.037798 (+ 2646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":283,"threads_started":1}
W20260806 04:52:31.038214 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:31.030041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:45631 (local address 127.13.148.190:42603)
0806 04:52:31.030192 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:31.030196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:31.030650 (+ 454us) server_negotiation.cc:407] Connection header received
0806 04:52:31.030774 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:31.030777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:31.030832 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:31.030904 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:31.031586 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:31.032463 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:31.033951 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:31.034370 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:31.034461 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:31.037998 (+ 3537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:45631: BlockingRecv error: recv got EOF from 127.13.148.130:45631 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 06 04:52:31 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:31 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:31.446449 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:31.438218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:53511)
0806 04:52:31.438864 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:31.438882 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:31.439041 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:31.439384 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:31.439388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:31.439883 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:31.440040 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:31.440049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:31.441052 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:31.441057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:31.441918 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:31.441928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:31.442699 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:31.442705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:31.442869 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:31.443456 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:31.443471 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:31.446130 (+ 2659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":353,"threads_started":1}
W20260806 04:52:31.446561 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:31.438422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:53511 (local address 127.13.148.190:42603)
0806 04:52:31.438677 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:31.438684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:31.438962 (+ 278us) server_negotiation.cc:407] Connection header received
0806 04:52:31.439171 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:31.439182 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:31.439245 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:31.439344 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:31.440179 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:31.440920 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:31.442076 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:31.442515 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:31.442655 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:31.446322 (+ 3667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.131:53511: BlockingRecv error: recv got EOF from 127.13.148.131:53511 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 06 04:52:31 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:31 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:31.600891 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:31.593178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:42815)
0806 04:52:31.593806 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:31.593822 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:31.593928 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:31.594202 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:31.594204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:31.594597 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:31.594749 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:31.594756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:31.595658 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:31.595661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:31.596525 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:31.596534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:31.597269 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:31.597273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:31.597394 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:31.598031 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:31.598047 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:31.600600 (+ 2553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":358,"threads_started":1}
W20260806 04:52:31.601151 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:31.593224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:42815 (local address 127.13.148.190:42603)
0806 04:52:31.593409 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:31.593413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:31.593889 (+ 476us) server_negotiation.cc:407] Connection header received
0806 04:52:31.594019 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:31.594023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:31.594084 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:31.594160 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:31.594856 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:31.595551 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:31.596640 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:31.597140 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:31.597233 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:31.600862 (+ 3629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:42815: BlockingRecv error: recv got EOF from 127.13.148.129:42815 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 06 04:52:32 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:32 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:32.047791 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:32.039181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:43783)
0806 04:52:32.039731 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:32.039750 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:32.039914 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:32.040275 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:32.040279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:32.040800 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:32.041014 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:32.041025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:32.041995 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:32.042000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:32.043528 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:32.043541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:32.044153 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:32.044158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:32.044313 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:32.044890 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:32.044910 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:32.047594 (+ 2684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":285,"threads_started":1}
W20260806 04:52:32.047950 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:32.039227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:43783 (local address 127.13.148.190:42603)
0806 04:52:32.039473 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:32.039483 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:32.039837 (+ 354us) server_negotiation.cc:407] Connection header received
0806 04:52:32.040036 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:32.040039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:32.040127 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:32.040218 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:32.041141 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:32.041871 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:32.043651 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:32.044008 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:32.044087 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:32.047745 (+ 3658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:43783: BlockingRecv error: recv got EOF from 127.13.148.130:43783 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 06 04:52:32 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:32 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:32.455068 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:32.447401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:38635)
0806 04:52:32.448002 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:32.448018 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:32.448127 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:32.448382 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:32.448385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:32.448827 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:32.449012 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:32.449021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:32.449917 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:32.449920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:32.450810 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:32.450820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:32.451426 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:32.451430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:32.451557 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:32.452094 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:32.452111 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:32.454846 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":331,"threads_started":1}
W20260806 04:52:32.455228 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:32.447446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:38635 (local address 127.13.148.190:42603)
0806 04:52:32.447617 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:32.447622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:32.448087 (+ 465us) server_negotiation.cc:407] Connection header received
0806 04:52:32.448206 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:32.448209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:32.448266 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:32.448340 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:32.449145 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:32.449815 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:32.450921 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:32.451286 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:32.451389 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:32.455027 (+ 3638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.131:38635: BlockingRecv error: recv got EOF from 127.13.148.131:38635 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 04:52:32 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:32 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:32.609589 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:32.601807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:43049)
0806 04:52:32.602346 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:32.602361 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:32.602477 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:32.602768 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:32.602771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:32.603124 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:32.603266 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:32.603274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:32.604185 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:32.604187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:32.605070 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:32.605080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:32.605691 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:32.605695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:32.605800 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:32.606320 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:32.606336 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:32.609250 (+ 2914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":303,"threads_started":1}
W20260806 04:52:32.609763 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:32.601901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:43049 (local address 127.13.148.190:42603)
0806 04:52:32.602079 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:32.602084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:32.602434 (+ 350us) server_negotiation.cc:407] Connection header received
0806 04:52:32.602577 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:32.602580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:32.602641 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:32.602717 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:32.603408 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:32.604081 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:32.605175 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:32.605550 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:32.605649 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:32.609513 (+ 3864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:43049: BlockingRecv error: recv got EOF from 127.13.148.129:43049 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 04:52:33 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:33 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:33.056473 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:33.048896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:37211)
0806 04:52:33.049375 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:33.049399 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:33.049500 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:33.049752 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:33.049755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:33.050115 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:33.050245 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:33.050252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:33.051417 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:33.051425 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:33.052535 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:33.052548 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:33.053224 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:33.053228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:33.053338 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:33.053872 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:33.053887 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:33.056274 (+ 2387us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":278,"threads_started":1}
W20260806 04:52:33.056685 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:33.048956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:37211 (local address 127.13.148.190:42603)
0806 04:52:33.049100 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:33.049104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:33.049464 (+ 360us) server_negotiation.cc:407] Connection header received
0806 04:52:33.049585 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:33.049588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:33.049642 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:33.049715 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:33.050414 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:33.051294 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:33.052661 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:33.053094 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:33.053182 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:33.056447 (+ 3265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:37211: BlockingRecv error: recv got EOF from 127.13.148.130:37211 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 06 04:52:33 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:33 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:33.464936 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:33.456044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:56445)
0806 04:52:33.456497 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:33.456512 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:33.456621 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:33.456924 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:33.456927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:33.457484 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:33.457718 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:33.457735 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:33.458937 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:33.458941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:33.459853 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:33.459863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:33.460742 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:33.460748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:33.461118 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:33.461731 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:33.461745 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:33.464622 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":233,"threads_started":1}
W20260806 04:52:33.465054 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:33.456255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:56445 (local address 127.13.148.190:42603)
0806 04:52:33.456503 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:33.456514 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:33.456562 (+ 48us) server_negotiation.cc:407] Connection header received
0806 04:52:33.456729 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:33.456732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:33.456808 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:33.456883 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:33.457931 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:33.458816 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:33.459994 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:33.460591 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:33.460703 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:33.464828 (+ 4125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.131:56445: BlockingRecv error: recv got EOF from 127.13.148.131:56445 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Aug 06 04:52:33 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:33 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:33.618434 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:33.610600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:50533)
0806 04:52:33.611097 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:33.611112 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:33.611216 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:33.611508 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:33.611512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:33.611960 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:33.612129 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:33.612137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:33.613440 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:33.613444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:33.614250 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:33.614260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:33.614874 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:33.614877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:33.614998 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:33.615523 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:33.615538 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:33.618217 (+ 2679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":266,"threads_started":1}
W20260806 04:52:33.618598 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:33.610661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:50533 (local address 127.13.148.190:42603)
0806 04:52:33.610841 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:33.610845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:33.611183 (+ 338us) server_negotiation.cc:407] Connection header received
0806 04:52:33.611307 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:33.611310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:33.611366 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:33.611439 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:33.612236 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:33.613316 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:33.614363 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:33.614756 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:33.614851 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:33.618392 (+ 3541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:50533: BlockingRecv error: recv got EOF from 127.13.148.129:50533 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 04:52:34 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:34 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:34.065392 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:34.057421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:51015)
0806 04:52:34.057944 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:34.057959 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:34.058104 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:34.058449 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:34.058453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:34.058898 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:34.059069 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:34.059077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:34.060004 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:34.060010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:34.060875 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:34.060886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:34.061542 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:34.061547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:34.061748 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:34.062374 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:34.062392 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:34.065184 (+ 2792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":274,"threads_started":1}
W20260806 04:52:34.065569 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:34.057554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:51015 (local address 127.13.148.190:42603)
0806 04:52:34.057780 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:34.057787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:34.058042 (+ 255us) server_negotiation.cc:407] Connection header received
0806 04:52:34.058239 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:34.058248 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:34.058309 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:34.058421 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:34.059204 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:34.059878 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:34.061014 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:34.061406 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:34.061527 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:34.065353 (+ 3826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:51015: BlockingRecv error: recv got EOF from 127.13.148.130:51015 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 04:52:34 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:34 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:34.473061 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:34.465735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:54745)
0806 04:52:34.466148 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:34.466163 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:34.466264 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:34.466526 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:34.466529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:34.466887 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:34.467050 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:34.467058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:34.467892 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:34.467895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:34.468699 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:34.468707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:34.469290 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:34.469294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:34.469401 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:34.469905 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:34.469919 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:34.472663 (+ 2744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":229,"threads_started":1}
W20260806 04:52:34.473129 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:34.465786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:54745 (local address 127.13.148.190:42603)
0806 04:52:34.465981 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:34.465989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:34.466235 (+ 246us) server_negotiation.cc:407] Connection header received
0806 04:52:34.466358 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:34.466361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:34.466418 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:34.466492 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:34.467163 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:34.467799 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:34.468842 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:34.469179 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:34.469259 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:34.472889 (+ 3630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.131:54745: BlockingRecv error: recv got EOF from 127.13.148.131:54745 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 04:52:34 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:34 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:34.627053 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:34.619292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:54701)
0806 04:52:34.619826 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:34.619841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:34.619945 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:34.620225 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:34.620228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:34.620602 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:34.620745 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:34.620753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:34.621695 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:34.621699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:34.622570 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:34.622581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:34.623258 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:34.623262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:34.623392 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:34.623956 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:34.623972 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:34.626736 (+ 2764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":328,"threads_started":1}
W20260806 04:52:34.627157 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:34.619362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:54701 (local address 127.13.148.190:42603)
0806 04:52:34.619544 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:34.619560 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:34.619910 (+ 350us) server_negotiation.cc:407] Connection header received
0806 04:52:34.620043 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:34.620046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:34.620103 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:34.620174 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:34.620918 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:34.621585 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:34.622691 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:34.623142 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:34.623235 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:34.626924 (+ 3689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:54701: BlockingRecv error: recv got EOF from 127.13.148.129:54701 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 04:52:35 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:35 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:35.079739 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:35.066272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:42629)
0806 04:52:35.066744 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:35.066758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:35.066902 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:35.067212 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:35.067216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:35.067582 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:35.067718 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:35.067725 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:35.068647 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:35.068655 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:35.069670 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:35.069682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:35.070769 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:35.070774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:35.070978 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:35.076618 (+ 5640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:35.076654 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:35.079433 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":267,"threads_started":1}
W20260806 04:52:35.079859 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:35.066318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:42629 (local address 127.13.148.190:42603)
0806 04:52:35.066535 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:35.066542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:35.066839 (+ 297us) server_negotiation.cc:407] Connection header received
0806 04:52:35.067026 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:35.067029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:35.067084 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:35.067173 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:35.067852 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:35.068512 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:35.069837 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:35.070635 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:35.070761 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:35.079646 (+ 8885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:42629: BlockingRecv error: recv got EOF from 127.13.148.130:42629 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 04:52:35 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:35 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:35.481959 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:35.474066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:41661)
0806 04:52:35.474576 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:35.474590 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:35.474694 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:35.474950 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:35.474952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:35.475286 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:35.475421 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:35.475427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:35.476398 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:35.476401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:35.477352 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:35.477363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:35.478051 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:35.478056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:35.478255 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:35.478855 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:35.478870 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:35.481714 (+ 2844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":296,"threads_started":1}
W20260806 04:52:35.482108 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:35.474123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:41661 (local address 127.13.148.190:42603)
0806 04:52:35.474270 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:35.474274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:35.474652 (+ 378us) server_negotiation.cc:407] Connection header received
0806 04:52:35.474775 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:35.474778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:35.474831 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:35.474900 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:35.475562 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:35.476278 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:35.477475 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:35.477889 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:35.478034 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:35.481881 (+ 3847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.131:41661: BlockingRecv error: recv got EOF from 127.13.148.131:41661 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 06 04:52:35 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:35 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:35.636106 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:35.628165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:44611)
0806 04:52:35.628740 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:35.628760 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:35.628943 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:35.629327 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:35.629331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:35.629859 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:35.630083 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:35.630097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:35.631003 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:35.631006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:35.631831 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:35.631841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:35.632467 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:35.632470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:35.632600 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:35.633211 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:35.633227 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:35.635821 (+ 2594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":303,"threads_started":1}
W20260806 04:52:35.636288 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:35.628228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:44611 (local address 127.13.148.190:42603)
0806 04:52:35.628464 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:35.628473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:35.628902 (+ 429us) server_negotiation.cc:407] Connection header received
0806 04:52:35.629086 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:35.629090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:35.629166 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:35.629263 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:35.630219 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:35.630888 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:35.631944 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:35.632322 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:35.632404 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:35.636040 (+ 3636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:44611: BlockingRecv error: recv got EOF from 127.13.148.129:44611 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 04:52:36 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:36 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:36.088747 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:36.080726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:34691)
0806 04:52:36.081297 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:36.081318 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:36.081499 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:36.081822 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:36.081826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:36.082210 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:36.082351 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:36.082359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:36.083535 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:36.083542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:36.084446 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:36.084456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:36.085222 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:36.085228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:36.085389 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:36.085888 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:36.085904 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:36.088427 (+ 2523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":372,"threads_started":1}
W20260806 04:52:36.088888 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:36.080954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:34691 (local address 127.13.148.190:42603)
0806 04:52:36.081136 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:36.081142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:36.081435 (+ 293us) server_negotiation.cc:407] Connection header received
0806 04:52:36.081629 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:36.081634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:36.081701 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:36.081788 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:36.082521 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:36.083386 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:36.084579 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:36.085038 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:36.085162 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:36.088611 (+ 3449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:34691: BlockingRecv error: recv got EOF from 127.13.148.130:34691 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 04:52:36 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:36 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:36.490978 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:36.482808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:60361)
0806 04:52:36.483376 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:36.483391 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:36.483548 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:36.483893 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:36.483898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:36.484278 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:36.484415 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:36.484423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:36.485350 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:36.485355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:36.486202 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:36.486212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:36.487081 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:36.487086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:36.487239 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:36.488033 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:36.488076 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:36.490683 (+ 2607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":282,"threads_started":1}
W20260806 04:52:36.491102 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:36.482926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:60361 (local address 127.13.148.190:42603)
0806 04:52:36.483146 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:36.483154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:36.483485 (+ 331us) server_negotiation.cc:407] Connection header received
0806 04:52:36.483690 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:36.483695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:36.483766 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:36.483864 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:36.484545 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:36.485227 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:36.486338 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:36.486897 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:36.487026 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:36.490875 (+ 3849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.131:60361: BlockingRecv error: recv got EOF from 127.13.148.131:60361 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 04:52:36 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:36 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:36.645117 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:36.637131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:50505)
0806 04:52:36.637599 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:36.637616 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:36.637730 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:36.638019 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:36.638021 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:36.638418 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:36.638640 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:36.638652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:36.640071 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:36.640075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:36.640930 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:36.640939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:36.641484 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:36.641487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:36.641587 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:36.642094 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:36.642108 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:36.644877 (+ 2769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":270,"threads_started":1}
W20260806 04:52:36.645259 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:36.637205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:50505 (local address 127.13.148.190:42603)
0806 04:52:36.637546 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:36.637552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:36.637695 (+ 143us) server_negotiation.cc:407] Connection header received
0806 04:52:36.637836 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:36.637839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:36.637908 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:36.637979 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:36.638803 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:36.639952 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:36.641031 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:36.641378 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:36.641465 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:36.645037 (+ 3572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:50505: BlockingRecv error: recv got EOF from 127.13.148.129:50505 (error 108)
Metrics: {"server-negotiator.queue_time_us":181}
Aug 06 04:52:37 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:37 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:37.097565 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:37.089782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:51029)
0806 04:52:37.090255 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:37.090269 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:37.090372 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:37.090600 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:37.090603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:37.090918 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:37.091080 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:37.091086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:37.092319 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:37.092325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:37.093359 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:37.093371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:37.094024 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:37.094028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:37.094131 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:37.094724 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:37.094740 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:37.097323 (+ 2583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":261,"threads_started":1}
W20260806 04:52:37.097707 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:37.089843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:51029 (local address 127.13.148.190:42603)
0806 04:52:37.090007 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:37.090010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:37.090332 (+ 322us) server_negotiation.cc:407] Connection header received
0806 04:52:37.090461 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:37.090464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:37.090518 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:37.090589 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:37.091184 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:37.092187 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:37.093493 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:37.093912 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:37.094002 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:37.097493 (+ 3491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:51029: BlockingRecv error: recv got EOF from 127.13.148.130:51029 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 04:52:37 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:37 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.131@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:37.499683 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:37.491739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.131:42355)
0806 04:52:37.492167 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:37.492181 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:37.492316 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:37.492663 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:37.492667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:37.493098 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:37.493270 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:37.493278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:37.494242 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:37.494248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:37.495045 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:37.495054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:37.495686 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:37.495690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:37.495848 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:37.496431 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:37.496447 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:37.499365 (+ 2918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":230,"threads_started":1}
W20260806 04:52:37.499794 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:37.491811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:42355 (local address 127.13.148.190:42603)
0806 04:52:37.492022 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:37.492030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:37.492258 (+ 228us) server_negotiation.cc:407] Connection header received
0806 04:52:37.492436 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:37.492461 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:37.492516 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:37.492623 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:37.493417 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:37.494113 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:37.495173 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:37.495556 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:37.495676 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:37.499589 (+ 3913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.131:42355: BlockingRecv error: recv got EOF from 127.13.148.131:42355 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 06 04:52:37 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:37 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.129@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:37.654174 23418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:37.646246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.129:56191)
0806 04:52:37.646756 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:37.646771 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:37.646870 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:37.647132 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:37.647135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:37.647492 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:37.647646 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:37.647654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:37.648555 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:37.648558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:37.649447 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:37.649457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:37.650069 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:37.650073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:37.650208 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:37.650803 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:37.650819 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:37.653789 (+ 2970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":278,"threads_started":1}
W20260806 04:52:37.654249 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:37.646270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:56191 (local address 127.13.148.190:42603)
0806 04:52:37.646437 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:37.646441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:37.646837 (+ 396us) server_negotiation.cc:407] Connection header received
0806 04:52:37.646957 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:37.646960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:37.647016 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:37.647088 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:37.647756 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:37.648429 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:37.649559 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:37.649955 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:37.650040 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:37.654007 (+ 3967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.129:56191: BlockingRecv error: recv got EOF from 127.13.148.129:56191 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 04:52:38 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:52:38 dist-test-slave-1x4g krb5kdc[22354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.130@KRBTEST.COM for oryx/127.13.148.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:52:38.106601 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:38.098572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42603 (local address 127.13.148.130:46393)
0806 04:52:38.099036 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:38.099050 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:38.099185 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:38.099498 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:38.099502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:38.099915 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:38.100082 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:38.100090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:38.101158 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:38.101164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:38.102019 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:38.102029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:38.102730 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:38.102735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:38.102898 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:38.103529 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:38.103546 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:38.106252 (+ 2706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.190:42603: Server oryx/127.13.148.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":237,"threads_started":1}
W20260806 04:52:38.106680 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:38.098632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:46393 (local address 127.13.148.190:42603)
0806 04:52:38.098835 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:38.098842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:38.099124 (+ 282us) server_negotiation.cc:407] Connection header received
0806 04:52:38.099316 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:38.099319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:38.099375 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:38.099474 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:38.100215 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:38.101023 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:38.102146 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:38.102571 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:38.102718 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:38.106437 (+ 3719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.130:46393: BlockingRecv error: recv got EOF from 127.13.148.130:46393 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 04:52:38.350659 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 22954
I20260806 04:52:38.361492 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:38.351526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45260 (local address 127.13.148.129:37501)
0806 04:52:38.352092 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:38.352098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:38.352123 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:52:38.352177 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:38.352183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:38.352393 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:38.352537 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:38.353932 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:38.354892 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:38.355947 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:38.356424 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:38.356516 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:38.358919 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:38.358938 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:38.358950 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:38.358974 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:38.361052 (+ 2078us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.148.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 04:52:38.361285 (+ 233us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45260: Request ticket server kudu/127.13.148.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":301,"threads_started":1}
W20260806 04:52:38.361618 23421 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45260: Request ticket server kudu/127.13.148.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 04:52:38.361716 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 22954: Not authorized: Client connection negotiation failed: client connection to 127.13.148.129:37501: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.148.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 04:52:38.361913 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 22954
I20260806 04:52:38.380273 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 23108
I20260806 04:52:38.390621 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:38.381016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.13.148.130:40231)
0806 04:52:38.381535 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:38.381541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:38.381570 (+ 29us) server_negotiation.cc:407] Connection header received
0806 04:52:38.381622 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:38.381628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:38.381837 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:38.381976 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:38.383062 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:38.384112 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:38.385318 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:38.385805 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:38.385895 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:38.387964 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:38.387972 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:38.387983 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:38.388011 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:38.390155 (+ 2144us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.148.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 04:52:38.390374 (+ 219us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46062: Request ticket server kudu/127.13.148.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":247,"threads_started":1}
W20260806 04:52:38.390666 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 23108: Not authorized: Client connection negotiation failed: client connection to 127.13.148.130:40231: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.148.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 04:52:38.390712 23422 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46062: Request ticket server kudu/127.13.148.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 04:52:38.390827 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 23108
I20260806 04:52:38.407943 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 23241
I20260806 04:52:38.418442 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:38.408737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.13.148.131:33797)
0806 04:52:38.409367 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:38.409375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:38.409404 (+ 29us) server_negotiation.cc:407] Connection header received
0806 04:52:38.409463 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:38.409469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:38.409690 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:38.409848 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:38.411012 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:38.411934 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:38.413035 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:38.413565 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:38.413661 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:38.415713 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:38.415720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:38.415731 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:38.415756 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:38.418028 (+ 2272us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.148.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 04:52:38.418246 (+ 218us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40320: Request ticket server kudu/127.13.148.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":341,"threads_started":1}
W20260806 04:52:38.418555 23423 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40320: Request ticket server kudu/127.13.148.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 04:52:38.418550 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 23241: Not authorized: Client connection negotiation failed: client connection to 127.13.148.131:33797: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.148.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 04:52:38.418771 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 23241
I20260806 04:52:38.435499 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 22885
I20260806 04:52:38.436553 22947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:38.544229 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 22885
2026-08-06T04:52:38Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20587 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T04:52:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:52:38Z Disabled control of system clock
I20260806 04:52:38.619983 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41735
--webserver_interface=127.13.148.190
--webserver_port=0
--builtin_ntp_servers=127.13.148.148:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:41735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 04:52:38.924208 23429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:38.924467 23429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:38.924542 23429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:38.930884 23429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:52:38.930975 23429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:38.931008 23429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:38.931051 23429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:52:38.931082 23429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:52:38.939288 23429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:41735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:38.940428 23429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:38.941989 23429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:38.952620 23434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:38.952667 23435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:38.952701 23437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:38.953466 23429 server_base.cc:1034] running on GCE node
I20260806 04:52:38.954319 23429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:38.955556 23429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:38.956830 23429 hybrid_clock.cc:648] HybridClock initialized: now 1785991958956741 us; error 57 us; skew 500 ppm
I20260806 04:52:38.957170 23429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:52:38.959110 23429 webserver.cc:460] Webserver started at http://127.13.148.190:43637/ using document root <none> and password file <none>
I20260806 04:52:38.959705 23429 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:38.959821 23429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:38.960127 23429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:38.962689 23429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "92ea95b69b344082895ade7a70d70276"
format_stamp: "Formatted at 2026-08-06 04:52:38 on dist-test-slave-1x4g"
server_key: "51412559ad1089af504c705fe5ba06ca"
server_key_iv: "3a888581835b8628e15c239e9922187b"
server_key_version: "encryptionkey@0"
I20260806 04:52:38.963480 23429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "92ea95b69b344082895ade7a70d70276"
format_stamp: "Formatted at 2026-08-06 04:52:38 on dist-test-slave-1x4g"
server_key: "51412559ad1089af504c705fe5ba06ca"
server_key_iv: "3a888581835b8628e15c239e9922187b"
server_key_version: "encryptionkey@0"
I20260806 04:52:38.968101 23429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260806 04:52:38.971138 23443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:38.972266 23429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:52:38.972438 23429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "92ea95b69b344082895ade7a70d70276"
format_stamp: "Formatted at 2026-08-06 04:52:38 on dist-test-slave-1x4g"
server_key: "51412559ad1089af504c705fe5ba06ca"
server_key_iv: "3a888581835b8628e15c239e9922187b"
server_key_version: "encryptionkey@0"
I20260806 04:52:38.972635 23429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:39.006152 23429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:39.006892 23429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:39.007181 23429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:39.024247 23429 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:41735
I20260806 04:52:39.024277 23494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:41735 every 8 connection(s)
I20260806 04:52:39.025497 23429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:52:39.027580 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 23429
I20260806 04:52:39.027781 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:52:39.028388 13906 external_mini_cluster.cc:1428] Setting key 7b6b0f73873aa3857a665a75cf902ce0
I20260806 04:52:39.029088 23495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:39.041024 23495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276: Bootstrap starting.
I20260806 04:52:39.042001 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:39.030312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.13.148.190:41735)
0806 04:52:39.031212 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:39.031223 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:39.031262 (+ 39us) server_negotiation.cc:407] Connection header received
0806 04:52:39.033664 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:39.033684 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:39.034123 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:39.034496 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:39.035050 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:39.036313 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:39.037764 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:39.038459 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:39.038726 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:39.039899 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:39.039906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:39.039917 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:52:39.040398 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:39.040485 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:52:39.040505 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:39.040632 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:39.040850 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 04:52:39.041115 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":284,"threads_started":1}
I20260806 04:52:39.044341 23495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:39.045503 23495 log.cc:824] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276: Log is configured to *not* fsync() on all Append() calls
I20260806 04:52:39.047879 23495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276: No bootstrap required, opened a new log
I20260806 04:52:39.055126 23495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ea95b69b344082895ade7a70d70276" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41735 } }
I20260806 04:52:39.055449 23495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:39.055560 23495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ea95b69b344082895ade7a70d70276, State: Initialized, Role: FOLLOWER
I20260806 04:52:39.056111 23495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ea95b69b344082895ade7a70d70276" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41735 } }
I20260806 04:52:39.056325 23495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:52:39.056459 23495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:52:39.056650 23495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:39.058881 23495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ea95b69b344082895ade7a70d70276" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41735 } }
I20260806 04:52:39.059342 23495 leader_election.cc:302] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92ea95b69b344082895ade7a70d70276; no voters:
I20260806 04:52:39.059840 23495 leader_election.cc:288] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:52:39.059988 23500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:52:39.060627 23500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [term 1 LEADER]: Becoming Leader. State: Replica: 92ea95b69b344082895ade7a70d70276, State: Running, Role: LEADER
I20260806 04:52:39.061088 23500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ea95b69b344082895ade7a70d70276" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41735 } }
I20260806 04:52:39.061707 23495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:52:39.063783 23502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92ea95b69b344082895ade7a70d70276. Latest consensus state: current_term: 1 leader_uuid: "92ea95b69b344082895ade7a70d70276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ea95b69b344082895ade7a70d70276" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41735 } } }
I20260806 04:52:39.063963 23501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92ea95b69b344082895ade7a70d70276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ea95b69b344082895ade7a70d70276" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41735 } } }
I20260806 04:52:39.064232 23502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:39.064280 23501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276 [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:39.066144 23507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:52:39.071115 23507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:52:39.079629 23507 catalog_manager.cc:1347] Generated new cluster ID: e6a77c024ef941b982431b8109801630
I20260806 04:52:39.079743 23507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:52:39.097792 23507 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:52:39.098965 23507 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:52:39.108464 23507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92ea95b69b344082895ade7a70d70276: Generated new TSK 0
I20260806 04:52:39.109424 23507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:52:39.124125 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41735
--builtin_ntp_servers=127.13.148.148:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 04:52:39.433813 23519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:39.434091 23519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:39.434166 23519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:39.440830 23519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:39.440928 23519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:39.441087 23519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:52:39.449105 23519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:39.450254 23519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:39.451908 23519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:39.463639 23524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:39.463810 23525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:39.464046 23527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:39.464702 23519 server_base.cc:1034] running on GCE node
I20260806 04:52:39.465265 23519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:39.466171 23519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:39.467384 23519 hybrid_clock.cc:648] HybridClock initialized: now 1785991959467335 us; error 58 us; skew 500 ppm
I20260806 04:52:39.467655 23519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:52:39.469636 23519 webserver.cc:460] Webserver started at http://127.13.148.129:39463/ using document root <none> and password file <none>
I20260806 04:52:39.470242 23519 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:39.470366 23519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:39.470660 23519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:39.473310 23519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "5f21cbbf07104788973e37e0f94b3022"
format_stamp: "Formatted at 2026-08-06 04:52:39 on dist-test-slave-1x4g"
server_key: "091e6966d70c449b256becd230215821"
server_key_iv: "adee0b949c9e5c5089d74d8c2395458e"
server_key_version: "encryptionkey@0"
I20260806 04:52:39.474089 23519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "5f21cbbf07104788973e37e0f94b3022"
format_stamp: "Formatted at 2026-08-06 04:52:39 on dist-test-slave-1x4g"
server_key: "091e6966d70c449b256becd230215821"
server_key_iv: "adee0b949c9e5c5089d74d8c2395458e"
server_key_version: "encryptionkey@0"
I20260806 04:52:39.478778 23519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:52:39.481773 23533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:39.482944 23519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:52:39.483129 23519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "5f21cbbf07104788973e37e0f94b3022"
format_stamp: "Formatted at 2026-08-06 04:52:39 on dist-test-slave-1x4g"
server_key: "091e6966d70c449b256becd230215821"
server_key_iv: "adee0b949c9e5c5089d74d8c2395458e"
server_key_version: "encryptionkey@0"
I20260806 04:52:39.483342 23519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:39.500535 23519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:39.501322 23519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:39.501616 23519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:39.502406 23519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:39.503773 23519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:39.503919 23519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:39.504058 23519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:39.504136 23519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:39.534221 23519 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:38103
I20260806 04:52:39.534255 23646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:38103 every 8 connection(s)
I20260806 04:52:39.535377 23519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:52:39.537742 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 23519
I20260806 04:52:39.537981 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:52:39.538590 13906 external_mini_cluster.cc:1428] Setting key 2334434cfd266eb10f41c6f81a0b720b
I20260806 04:52:39.541199 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41735
--builtin_ntp_servers=127.13.148.148:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 04:52:39.549307 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:39.537934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:43001 (local address 127.13.148.190:41735)
0806 04:52:39.538134 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:39.538140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:39.539624 (+ 1484us) server_negotiation.cc:407] Connection header received
0806 04:52:39.541407 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:39.541415 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:39.541510 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:39.541574 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:39.542959 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:39.544090 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:39.546140 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:39.546775 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:39.546903 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:39.548355 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:39.548362 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:39.548372 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:52:39.548393 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:39.548429 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:52:39.548443 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:39.548506 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:39.548807 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 04:52:39.548911 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 04:52:39.550227 23649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:39.538460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41735 (local address 127.13.148.129:43001)
0806 04:52:39.539471 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:39.539512 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:39.541083 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:39.542003 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:39.542014 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:39.542109 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:39.542758 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:39.542798 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:39.544221 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:39.544227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:39.546004 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:39.546018 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:39.546984 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:39.546991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:39.547275 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:39.548013 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:52:39.548018 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:39.548074 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:52:39.548157 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:39.548575 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:39.548586 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:39.548594 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 04:52:39.548735 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 04:52:39.549117 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":358,"threads_started":1}
I20260806 04:52:39.551573 23647 heartbeater.cc:347] Connected to a master server at 127.13.148.190:41735
I20260806 04:52:39.551846 23647 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:39.552639 23647 heartbeater.cc:511] Master 127.13.148.190:41735 requested a full tablet report, sending...
I20260806 04:52:39.554731 23460 ts_manager.cc:194] Registered new tserver with Master: 5f21cbbf07104788973e37e0f94b3022 (127.13.148.129:38103)
I20260806 04:52:39.556041 23460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.129:43001
W20260806 04:52:39.846262 23650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:39.846495 23650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:39.846571 23650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:39.853018 23650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:39.853104 23650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:39.853271 23650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:52:39.861826 23650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:39.863070 23650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:39.864774 23650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:39.877417 23655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:39.877537 23658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:39.877513 23656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:39.878352 23650 server_base.cc:1034] running on GCE node
I20260806 04:52:39.878819 23650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:39.879585 23650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:39.880813 23650 hybrid_clock.cc:648] HybridClock initialized: now 1785991959880726 us; error 33 us; skew 500 ppm
I20260806 04:52:39.881108 23650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:52:39.883121 23650 webserver.cc:460] Webserver started at http://127.13.148.130:38659/ using document root <none> and password file <none>
I20260806 04:52:39.883728 23650 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:39.883808 23650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:39.884150 23650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:39.887045 23650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "c48a8ee6f3e2495eb5505dd3d4522a6a"
format_stamp: "Formatted at 2026-08-06 04:52:39 on dist-test-slave-1x4g"
server_key: "2f4788526b855090a2fbe052e454e5d0"
server_key_iv: "2286363e758acee52f0d5a834dd1e65a"
server_key_version: "encryptionkey@0"
I20260806 04:52:39.887790 23650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "c48a8ee6f3e2495eb5505dd3d4522a6a"
format_stamp: "Formatted at 2026-08-06 04:52:39 on dist-test-slave-1x4g"
server_key: "2f4788526b855090a2fbe052e454e5d0"
server_key_iv: "2286363e758acee52f0d5a834dd1e65a"
server_key_version: "encryptionkey@0"
I20260806 04:52:39.892293 23650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260806 04:52:39.895371 23664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:39.896481 23650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:52:39.896626 23650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "c48a8ee6f3e2495eb5505dd3d4522a6a"
format_stamp: "Formatted at 2026-08-06 04:52:39 on dist-test-slave-1x4g"
server_key: "2f4788526b855090a2fbe052e454e5d0"
server_key_iv: "2286363e758acee52f0d5a834dd1e65a"
server_key_version: "encryptionkey@0"
I20260806 04:52:39.896860 23650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:39.935369 23650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:39.936224 23650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:39.936476 23650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:39.937438 23650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:39.938810 23650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:39.938916 23650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:39.938997 23650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:39.939050 23650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:39.969659 23650 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:33559
I20260806 04:52:39.969735 23777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:33559 every 8 connection(s)
I20260806 04:52:39.970853 23650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:52:39.976454 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 23650
I20260806 04:52:39.976667 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:52:39.977250 13906 external_mini_cluster.cc:1428] Setting key 056da27841af7aba88d1ca78ce7ecffa
I20260806 04:52:39.980084 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41735
--builtin_ntp_servers=127.13.148.148:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 04:52:39.984097 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:39.973391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:41815 (local address 127.13.148.190:41735)
0806 04:52:39.973622 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:39.973628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:39.974947 (+ 1319us) server_negotiation.cc:407] Connection header received
0806 04:52:39.977002 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:39.977008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:39.977096 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:39.977204 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:39.978279 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:39.979242 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:39.981481 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:39.981809 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:39.981893 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:39.983434 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:39.983445 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:39.983449 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:52:39.983465 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:39.983493 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:52:39.983505 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:39.983571 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:39.983810 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 04:52:39.983882 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 04:52:39.985554 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:39.973926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41735 (local address 127.13.148.130:41815)
0806 04:52:39.974796 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:39.974830 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:39.976661 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:39.977427 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:39.977435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:39.977519 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:39.978115 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:39.978137 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:39.979407 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:39.979412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:39.981335 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:39.981353 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:39.982011 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:39.982019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:39.982308 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:39.983099 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:52:39.983105 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:39.983154 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:52:39.983252 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:39.983598 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:39.983610 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:39.983621 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 04:52:39.983797 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 04:52:39.984175 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":265,"threads_started":1}
I20260806 04:52:39.986959 23778 heartbeater.cc:347] Connected to a master server at 127.13.148.190:41735
I20260806 04:52:39.987262 23778 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:39.987969 23778 heartbeater.cc:511] Master 127.13.148.190:41735 requested a full tablet report, sending...
I20260806 04:52:39.989276 23460 ts_manager.cc:194] Registered new tserver with Master: c48a8ee6f3e2495eb5505dd3d4522a6a (127.13.148.130:33559)
I20260806 04:52:39.990015 23460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.130:41815
W20260806 04:52:40.292191 23781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:40.292451 23781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:40.292537 23781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:40.298812 23781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:40.298887 23781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:40.298974 23781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:52:40.306982 23781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:40.308049 23781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:40.309682 23781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:40.321666 23786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:40.321621 23787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:40.321869 23789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:40.322515 23781 server_base.cc:1034] running on GCE node
I20260806 04:52:40.323200 23781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:40.324172 23781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:40.325434 23781 hybrid_clock.cc:648] HybridClock initialized: now 1785991960325358 us; error 73 us; skew 500 ppm
I20260806 04:52:40.325697 23781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:52:40.332412 23781 webserver.cc:460] Webserver started at http://127.13.148.131:37367/ using document root <none> and password file <none>
I20260806 04:52:40.333120 23781 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:40.333212 23781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:40.333488 23781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:40.336190 23781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "56acfcdec08442b78ba8f9308cb743eb"
format_stamp: "Formatted at 2026-08-06 04:52:40 on dist-test-slave-1x4g"
server_key: "c637e5def9a7226217711dd1f981b527"
server_key_iv: "98a01b50b122372877d82818751898f3"
server_key_version: "encryptionkey@0"
I20260806 04:52:40.337035 23781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "56acfcdec08442b78ba8f9308cb743eb"
format_stamp: "Formatted at 2026-08-06 04:52:40 on dist-test-slave-1x4g"
server_key: "c637e5def9a7226217711dd1f981b527"
server_key_iv: "98a01b50b122372877d82818751898f3"
server_key_version: "encryptionkey@0"
I20260806 04:52:40.341643 23781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:52:40.344746 23795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:40.345988 23781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:52:40.346151 23781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "56acfcdec08442b78ba8f9308cb743eb"
format_stamp: "Formatted at 2026-08-06 04:52:40 on dist-test-slave-1x4g"
server_key: "c637e5def9a7226217711dd1f981b527"
server_key_iv: "98a01b50b122372877d82818751898f3"
server_key_version: "encryptionkey@0"
I20260806 04:52:40.346345 23781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:40.363679 23781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:40.364416 23781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:40.364691 23781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:40.365533 23781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:40.366936 23781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:40.367062 23781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:40.367168 23781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:40.367250 23781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:40.398506 23781 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:35813
I20260806 04:52:40.398571 23910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:35813 every 8 connection(s)
I20260806 04:52:40.399943 23781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:52:40.405704 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 23781
I20260806 04:52:40.405918 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:52:40.406415 13906 external_mini_cluster.cc:1428] Setting key ec1dcff4d38d08483d5b37fbd3ab9f0d
I20260806 04:52:40.412508 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:40.402588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:44679 (local address 127.13.148.190:41735)
0806 04:52:40.402769 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:40.402775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:40.404119 (+ 1344us) server_negotiation.cc:407] Connection header received
0806 04:52:40.405850 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:40.405855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:40.405952 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:40.406067 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:40.407253 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:40.408386 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:40.410230 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:40.410631 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:40.410725 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:40.411915 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:40.411919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:40.411922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:52:40.411937 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:40.411972 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:52:40.411984 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:40.412046 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:40.412224 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 04:52:40.412297 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:52:40.413489 23913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:40.403078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41735 (local address 127.13.148.131:44679)
0806 04:52:40.403945 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:40.404002 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:40.405550 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:40.406269 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:40.406276 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:40.406358 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:40.407062 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:40.407090 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:40.408540 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:40.408543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:40.410099 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:40.410115 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:40.410806 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:40.410810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:40.411038 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:40.411653 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:52:40.411655 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:40.411691 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:52:40.411759 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:40.412060 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:40.412067 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:40.412072 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 04:52:40.412196 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 04:52:40.412452 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":256,"threads_started":1}
I20260806 04:52:40.414839 23911 heartbeater.cc:347] Connected to a master server at 127.13.148.190:41735
I20260806 04:52:40.415081 23911 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:40.415692 23911 heartbeater.cc:511] Master 127.13.148.190:41735 requested a full tablet report, sending...
I20260806 04:52:40.417035 23460 ts_manager.cc:194] Registered new tserver with Master: 56acfcdec08442b78ba8f9308cb743eb (127.13.148.131:35813)
I20260806 04:52:40.417763 23460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.131:44679
I20260806 04:52:40.421543 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 04:52:40.427335 23498 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:40.425791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.13.148.190:41735)
0806 04:52:40.425984 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:40.425992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:40.426426 (+ 434us) server_negotiation.cc:407] Connection header received
0806 04:52:40.426548 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:40.426552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:40.426622 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:40.426705 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:40.427126 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33544: BlockingRecv error: recv got EOF from 127.0.0.1:33544 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:52:40.428078 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 23519
I20260806 04:52:40.435564 23920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:40.428960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.13.148.129:38103)
0806 04:52:40.429490 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:40.429496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:40.429520 (+ 24us) server_negotiation.cc:407] Connection header received
0806 04:52:40.429571 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:40.429576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:40.429761 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:40.429873 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:40.430217 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:40.431146 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:40.432862 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:40.433387 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:40.433497 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:40.434387 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:40.434394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:40.434405 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:52:40.434919 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:40.434962 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:52:40.434980 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:40.435078 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:40.435188 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 04:52:40.435287 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":254,"threads_started":1}
I20260806 04:52:40.436728 23642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:40.510527 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 23519
I20260806 04:52:40.528115 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 23650
I20260806 04:52:40.535010 23922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:40.529022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.13.148.130:33559)
0806 04:52:40.529496 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:40.529503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:40.529528 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:52:40.529583 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:40.529589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:40.529823 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:40.529935 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:40.530252 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:40.531103 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:40.532114 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:40.532590 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:40.532672 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:40.533752 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:40.533760 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:40.533772 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:52:40.534337 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:40.534377 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:52:40.534394 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:40.534487 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:40.534581 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 04:52:40.534708 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":226,"threads_started":1}
I20260806 04:52:40.536073 23773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:40.610843 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 23650
I20260806 04:52:40.627671 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 23781
I20260806 04:52:40.635505 23924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:40.628700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.13.148.131:35813)
0806 04:52:40.629363 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:40.629370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:40.629398 (+ 28us) server_negotiation.cc:407] Connection header received
0806 04:52:40.629461 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:40.629467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:40.629674 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:40.629792 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:40.630175 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:40.631184 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:40.632294 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:40.633056 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:40.633152 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:40.634128 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:40.634136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:40.634150 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:52:40.634727 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:40.634770 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:52:40.634788 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:40.634869 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:40.634990 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 04:52:40.635146 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":388,"threads_started":1}
I20260806 04:52:40.636749 23906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:40.714991 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 23781
I20260806 04:52:40.732961 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 23429
I20260806 04:52:40.734230 23490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:40.835115 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 23429
2026-08-06T04:52:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2291 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T04:52:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:52:40Z Disabled control of system clock
I20260806 04:52:40.913635 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:33061
--webserver_interface=127.13.148.190
--webserver_port=0
--builtin_ntp_servers=127.13.148.148:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:33061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 04:52:41.215715 23930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:41.215945 23930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:41.216017 23930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:41.222612 23930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:52:41.222698 23930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:41.222733 23930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:41.222800 23930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:52:41.222857 23930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:52:41.230943 23930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:33061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:33061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:41.231978 23930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:41.233589 23930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:41.244457 23935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:41.244534 23936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:41.244494 23938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:41.245337 23930 server_base.cc:1034] running on GCE node
I20260806 04:52:41.246069 23930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:41.247257 23930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:41.248466 23930 hybrid_clock.cc:648] HybridClock initialized: now 1785991961248428 us; error 48 us; skew 500 ppm
I20260806 04:52:41.248761 23930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:52:41.250722 23930 webserver.cc:460] Webserver started at http://127.13.148.190:39841/ using document root <none> and password file <none>
I20260806 04:52:41.251325 23930 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:41.251443 23930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:41.251732 23930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:41.254343 23930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "03c8ba9e73a14650a69820ab1de04fe4"
format_stamp: "Formatted at 2026-08-06 04:52:41 on dist-test-slave-1x4g"
server_key: "59ff4bb1d3a9cb634ee92cbb43cebb4a"
server_key_iv: "e3f7d1e711e901ffc51179c6ba9cc076"
server_key_version: "encryptionkey@0"
I20260806 04:52:41.255072 23930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "03c8ba9e73a14650a69820ab1de04fe4"
format_stamp: "Formatted at 2026-08-06 04:52:41 on dist-test-slave-1x4g"
server_key: "59ff4bb1d3a9cb634ee92cbb43cebb4a"
server_key_iv: "e3f7d1e711e901ffc51179c6ba9cc076"
server_key_version: "encryptionkey@0"
I20260806 04:52:41.259771 23930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260806 04:52:41.262648 23944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:41.263857 23930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:52:41.264067 23930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "03c8ba9e73a14650a69820ab1de04fe4"
format_stamp: "Formatted at 2026-08-06 04:52:41 on dist-test-slave-1x4g"
server_key: "59ff4bb1d3a9cb634ee92cbb43cebb4a"
server_key_iv: "e3f7d1e711e901ffc51179c6ba9cc076"
server_key_version: "encryptionkey@0"
I20260806 04:52:41.264272 23930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:41.269194 23930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:41.269827 23930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:41.270089 23930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:41.286828 23930 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:33061
I20260806 04:52:41.286854 23997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:33061 every 8 connection(s)
I20260806 04:52:41.287995 23930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:52:41.290988 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 23930
I20260806 04:52:41.291172 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:52:41.291855 23998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:41.291934 13906 external_mini_cluster.cc:1428] Setting key 73d5619bf983e14964c3069169e49160
I20260806 04:52:41.299119 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:41.293822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.13.148.190:33061)
0806 04:52:41.294759 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:41.294768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:41.294808 (+ 40us) server_negotiation.cc:407] Connection header received
0806 04:52:41.296041 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:41.296060 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:41.296397 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:41.296726 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:41.297779 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:41.297785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:41.297797 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:52:41.297809 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:41.297880 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:52:41.297893 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:41.298046 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:41.298127 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 04:52:41.298375 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":396,"threads_started":1}
I20260806 04:52:41.303675 23998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4: Bootstrap starting.
I20260806 04:52:41.306329 23998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:41.307312 23998 log.cc:824] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4: Log is configured to *not* fsync() on all Append() calls
I20260806 04:52:41.309656 23998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4: No bootstrap required, opened a new log
I20260806 04:52:41.315848 23998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c8ba9e73a14650a69820ab1de04fe4" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33061 } }
I20260806 04:52:41.316145 23998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:41.316252 23998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03c8ba9e73a14650a69820ab1de04fe4, State: Initialized, Role: FOLLOWER
I20260806 04:52:41.316880 23998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c8ba9e73a14650a69820ab1de04fe4" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33061 } }
I20260806 04:52:41.317109 23998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:52:41.317240 23998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:52:41.317415 23998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:41.319388 23998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c8ba9e73a14650a69820ab1de04fe4" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33061 } }
I20260806 04:52:41.319828 23998 leader_election.cc:302] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03c8ba9e73a14650a69820ab1de04fe4; no voters:
I20260806 04:52:41.320350 23998 leader_election.cc:288] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:52:41.320446 24003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:52:41.321128 24003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [term 1 LEADER]: Becoming Leader. State: Replica: 03c8ba9e73a14650a69820ab1de04fe4, State: Running, Role: LEADER
I20260806 04:52:41.321663 24003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c8ba9e73a14650a69820ab1de04fe4" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33061 } }
I20260806 04:52:41.322162 23998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:52:41.324342 24004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03c8ba9e73a14650a69820ab1de04fe4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c8ba9e73a14650a69820ab1de04fe4" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33061 } } }
I20260806 04:52:41.324434 24005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03c8ba9e73a14650a69820ab1de04fe4. Latest consensus state: current_term: 1 leader_uuid: "03c8ba9e73a14650a69820ab1de04fe4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c8ba9e73a14650a69820ab1de04fe4" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33061 } } }
I20260806 04:52:41.324853 24004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:41.324854 24005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:41.326223 24009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:52:41.330344 24009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:52:41.338325 24009 catalog_manager.cc:1347] Generated new cluster ID: 55094746975043b79e9328c99d9d64c7
I20260806 04:52:41.338414 24009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:52:41.339910 24021 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 04:52:41.346623 24009 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:52:41.347007 24009 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:52:41.356734 24009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03c8ba9e73a14650a69820ab1de04fe4: Generated new TSK 0
I20260806 04:52:41.357582 24009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:52:41.367089 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33061
--builtin_ntp_servers=127.13.148.148:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 04:52:41.677404 24023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:41.677707 24023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:41.677891 24023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:41.684500 24023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:41.684617 24023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:41.684787 24023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:52:41.692689 24023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:41.693917 24023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:41.695608 24023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:41.707723 24028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:41.707752 24029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:41.707805 24032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:41.708499 24023 server_base.cc:1034] running on GCE node
I20260806 04:52:41.709110 24023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:41.710016 24023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:41.711222 24023 hybrid_clock.cc:648] HybridClock initialized: now 1785991961711168 us; error 45 us; skew 500 ppm
I20260806 04:52:41.711498 24023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:52:41.713625 24023 webserver.cc:460] Webserver started at http://127.13.148.129:45805/ using document root <none> and password file <none>
I20260806 04:52:41.714314 24023 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:41.714432 24023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:41.714777 24023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:41.717633 24023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "7a5eae357106407a971751235764400c"
format_stamp: "Formatted at 2026-08-06 04:52:41 on dist-test-slave-1x4g"
server_key: "401ab02d9029f8ebfe6f5a822f0e36d1"
server_key_iv: "e925b663e20784e53624f877081f2deb"
server_key_version: "encryptionkey@0"
I20260806 04:52:41.718396 24023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "7a5eae357106407a971751235764400c"
format_stamp: "Formatted at 2026-08-06 04:52:41 on dist-test-slave-1x4g"
server_key: "401ab02d9029f8ebfe6f5a822f0e36d1"
server_key_iv: "e925b663e20784e53624f877081f2deb"
server_key_version: "encryptionkey@0"
I20260806 04:52:41.723282 24023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 04:52:41.726421 24038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:41.727773 24023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:52:41.727936 24023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "7a5eae357106407a971751235764400c"
format_stamp: "Formatted at 2026-08-06 04:52:41 on dist-test-slave-1x4g"
server_key: "401ab02d9029f8ebfe6f5a822f0e36d1"
server_key_iv: "e925b663e20784e53624f877081f2deb"
server_key_version: "encryptionkey@0"
I20260806 04:52:41.728138 24023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:41.733366 24023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:41.734118 24023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:41.734409 24023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:41.735260 24023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:41.736518 24023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:41.736663 24023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:41.736873 24023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:41.736974 24023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 04:52:41.766420 24023 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:37063
I20260806 04:52:41.766500 24154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:37063 every 8 connection(s)
I20260806 04:52:41.767765 24023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:52:41.771735 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 24023
I20260806 04:52:41.771939 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:52:41.772806 13906 external_mini_cluster.cc:1428] Setting key 6a309a07ba03d2c1d44570a805241cfb
I20260806 04:52:41.775750 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33061
--builtin_ntp_servers=127.13.148.148:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 04:52:41.776217 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:41.770149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:33407 (local address 127.13.148.190:33061)
0806 04:52:41.770314 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:41.770318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:41.771654 (+ 1336us) server_negotiation.cc:407] Connection header received
0806 04:52:41.773987 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:41.773993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:41.774049 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:41.774176 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:41.775448 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:41.775453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:41.775457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:52:41.775478 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:41.775536 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:52:41.775554 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:41.775676 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:41.775849 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 04:52:41.775913 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:52:41.777143 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:41.770588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33061 (local address 127.13.148.129:33407)
0806 04:52:41.771479 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:41.771531 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:41.773689 (+ 2158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:41.774363 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:41.774371 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:41.774452 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:41.775162 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:52:41.775164 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:41.775197 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:52:41.775273 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:41.775654 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:41.775661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:41.775666 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 04:52:41.775893 (+ 227us) client_negotiation.cc:240] Negotiation successful
0806 04:52:41.776131 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":245,"threads_started":1}
I20260806 04:52:41.778594 24155 heartbeater.cc:347] Connected to a master server at 127.13.148.190:33061
I20260806 04:52:41.778843 24155 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:41.779407 24155 heartbeater.cc:511] Master 127.13.148.190:33061 requested a full tablet report, sending...
I20260806 04:52:41.781402 23963 ts_manager.cc:194] Registered new tserver with Master: 7a5eae357106407a971751235764400c (127.13.148.129:37063)
W20260806 04:52:42.084677 24158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:42.084995 24158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:42.085153 24158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:42.091205 24158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:42.091318 24158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:42.091481 24158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:52:42.099395 24158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:42.100519 24158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:42.102149 24158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:42.114315 24167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:42.114349 24165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:42.114276 24164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:42.115278 24158 server_base.cc:1034] running on GCE node
I20260806 04:52:42.115785 24158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:42.116688 24158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:42.117897 24158 hybrid_clock.cc:648] HybridClock initialized: now 1785991962117825 us; error 63 us; skew 500 ppm
I20260806 04:52:42.118173 24158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:52:42.120247 24158 webserver.cc:460] Webserver started at http://127.13.148.130:36727/ using document root <none> and password file <none>
I20260806 04:52:42.120956 24158 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:42.121073 24158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:42.121392 24158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:42.126132 24158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "7282341f745540818e9417318a2beadf"
format_stamp: "Formatted at 2026-08-06 04:52:42 on dist-test-slave-1x4g"
server_key: "fb6632c5add12376d6b49fe82f923cb7"
server_key_iv: "90a8566eabd89ecf4eebc8091239cb91"
server_key_version: "encryptionkey@0"
I20260806 04:52:42.126907 24158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "7282341f745540818e9417318a2beadf"
format_stamp: "Formatted at 2026-08-06 04:52:42 on dist-test-slave-1x4g"
server_key: "fb6632c5add12376d6b49fe82f923cb7"
server_key_iv: "90a8566eabd89ecf4eebc8091239cb91"
server_key_version: "encryptionkey@0"
I20260806 04:52:42.131554 24158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:52:42.134912 24174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:42.136096 24158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 04:52:42.136227 24158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "7282341f745540818e9417318a2beadf"
format_stamp: "Formatted at 2026-08-06 04:52:42 on dist-test-slave-1x4g"
server_key: "fb6632c5add12376d6b49fe82f923cb7"
server_key_iv: "90a8566eabd89ecf4eebc8091239cb91"
server_key_version: "encryptionkey@0"
I20260806 04:52:42.136389 24158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:42.141345 24158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:42.142023 24158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:42.142242 24158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:42.143029 24158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:42.144282 24158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:42.144357 24158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:42.144425 24158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:42.144467 24158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:42.175259 24158 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:33283
I20260806 04:52:42.175336 24292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:33283 every 8 connection(s)
I20260806 04:52:42.176397 24158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:52:42.182332 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 24158
I20260806 04:52:42.182497 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:52:42.183059 13906 external_mini_cluster.cc:1428] Setting key d14c18ef87fb095cfc9eb5c205b8169d
I20260806 04:52:42.184855 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:42.179911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:46401 (local address 127.13.148.190:33061)
0806 04:52:42.180085 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:42.180091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:42.180615 (+ 524us) server_negotiation.cc:407] Connection header received
0806 04:52:42.182591 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:42.182596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:42.182663 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:42.182769 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:42.184160 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:42.184166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:42.184170 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:52:42.184199 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:42.184236 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:52:42.184250 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:42.184361 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:42.184502 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 04:52:42.184556 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 04:52:42.185695 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:42.179424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33061 (local address 127.13.148.130:46401)
0806 04:52:42.180462 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:42.180506 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:42.182300 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:42.182947 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:42.182955 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:42.183054 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:42.183802 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:52:42.183805 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:42.183852 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:52:42.183970 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:42.184335 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:42.184340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:42.184344 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 04:52:42.184526 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 04:52:42.184738 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":292,"threads_started":1}
I20260806 04:52:42.185909 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33061
--builtin_ntp_servers=127.13.148.148:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 04:52:42.186985 24293 heartbeater.cc:347] Connected to a master server at 127.13.148.190:33061
I20260806 04:52:42.187264 24293 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:42.187860 24293 heartbeater.cc:511] Master 127.13.148.190:33061 requested a full tablet report, sending...
I20260806 04:52:42.189162 23963 ts_manager.cc:194] Registered new tserver with Master: 7282341f745540818e9417318a2beadf (127.13.148.130:33283)
W20260806 04:52:42.491230 24296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:42.491468 24296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:42.491546 24296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:42.498154 24296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:42.498234 24296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:42.498345 24296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:52:42.506606 24296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:42.507706 24296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:42.509382 24296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:42.521167 24301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:42.521230 24302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:42.521200 24304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:42.522055 24296 server_base.cc:1034] running on GCE node
I20260806 04:52:42.522552 24296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:42.523468 24296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:42.524681 24296 hybrid_clock.cc:648] HybridClock initialized: now 1785991962524636 us; error 38 us; skew 500 ppm
I20260806 04:52:42.524974 24296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:52:42.527087 24296 webserver.cc:460] Webserver started at http://127.13.148.131:37243/ using document root <none> and password file <none>
I20260806 04:52:42.527696 24296 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:42.527812 24296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:42.528091 24296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:42.530671 24296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "dcd91c195c77403ba78cfce3d15440f4"
format_stamp: "Formatted at 2026-08-06 04:52:42 on dist-test-slave-1x4g"
server_key: "78aa5c1440403314506b0023bdbbe352"
server_key_iv: "4e512a7863bb7086eb1aafde8540f48f"
server_key_version: "encryptionkey@0"
I20260806 04:52:42.531370 24296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "dcd91c195c77403ba78cfce3d15440f4"
format_stamp: "Formatted at 2026-08-06 04:52:42 on dist-test-slave-1x4g"
server_key: "78aa5c1440403314506b0023bdbbe352"
server_key_iv: "4e512a7863bb7086eb1aafde8540f48f"
server_key_version: "encryptionkey@0"
I20260806 04:52:42.535794 24296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:52:42.538764 24310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:42.539829 24296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 04:52:42.539987 24296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "dcd91c195c77403ba78cfce3d15440f4"
format_stamp: "Formatted at 2026-08-06 04:52:42 on dist-test-slave-1x4g"
server_key: "78aa5c1440403314506b0023bdbbe352"
server_key_iv: "4e512a7863bb7086eb1aafde8540f48f"
server_key_version: "encryptionkey@0"
I20260806 04:52:42.540205 24296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:42.545281 24296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:42.545962 24296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:42.546204 24296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:42.546993 24296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:42.548285 24296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:42.548389 24296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:42.548511 24296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:42.548586 24296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:42.579020 24296 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:46241
I20260806 04:52:42.579083 24426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:46241 every 8 connection(s)
I20260806 04:52:42.580282 24296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:52:42.587747 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:42.582786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:39383 (local address 127.13.148.190:33061)
0806 04:52:42.583004 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:42.583008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:42.584346 (+ 1338us) server_negotiation.cc:407] Connection header received
0806 04:52:42.585998 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:42.586002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:42.586052 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:42.586132 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:42.587233 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:42.587237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:42.587240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:52:42.587253 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:42.587284 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:52:42.587294 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:42.587383 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:42.587523 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 04:52:42.587567 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 04:52:42.588536 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:42.583310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33061 (local address 127.13.148.131:39383)
0806 04:52:42.584201 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:42.584234 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:42.585763 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:42.586320 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:42.586327 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:42.586408 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:42.586985 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:52:42.586988 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:42.587021 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:52:42.587093 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:42.587370 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:42.587377 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:42.587382 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 04:52:42.587562 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 04:52:42.587722 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":266,"threads_started":1}
I20260806 04:52:42.588812 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 24296
I20260806 04:52:42.588999 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:52:42.589510 13906 external_mini_cluster.cc:1428] Setting key 5280763e6a6a193e7a412a099791c978
I20260806 04:52:42.589826 24427 heartbeater.cc:347] Connected to a master server at 127.13.148.190:33061
I20260806 04:52:42.590097 24427 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:42.590708 24427 heartbeater.cc:511] Master 127.13.148.190:33061 requested a full tablet report, sending...
I20260806 04:52:42.592120 23962 ts_manager.cc:194] Registered new tserver with Master: dcd91c195c77403ba78cfce3d15440f4 (127.13.148.131:46241)
I20260806 04:52:42.604225 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 04:52:42.609596 24001 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:52:42.608215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.13.148.190:33061)
0806 04:52:42.608387 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:42.608394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:42.608842 (+ 448us) server_negotiation.cc:407] Connection header received
0806 04:52:42.609000 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:42.609004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:42.609047 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:42.609116 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:42.609367 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50854: BlockingRecv error: recv got EOF from 127.0.0.1:50854 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:52:42.610070 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 24023
I20260806 04:52:42.613508 24436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:42.611012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44000 (local address 127.13.148.129:37063)
0806 04:52:42.611460 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:42.611466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:42.611489 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:52:42.611540 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:42.611545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:42.611728 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:42.611871 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:42.612749 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:42.612754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:42.612763 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:52:42.612836 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:42.612886 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:52:42.612901 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:42.613053 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:42.613156 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 04:52:42.613249 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":230,"threads_started":1}
I20260806 04:52:42.614614 24150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:42.690797 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 24023
I20260806 04:52:42.707720 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 24158
I20260806 04:52:42.710996 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:42.708599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.13.148.130:33283)
0806 04:52:42.709119 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:42.709126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:42.709149 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:52:42.709205 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:42.709210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:42.709409 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:42.709524 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:42.710395 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:42.710400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:42.710411 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:52:42.710426 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:42.710465 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:52:42.710479 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:42.710577 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:42.710682 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 04:52:42.710761 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":280,"threads_started":1}
I20260806 04:52:42.712101 24288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:42.785190 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 24158
I20260806 04:52:42.801761 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 24296
I20260806 04:52:42.805702 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:42.802645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.13.148.131:46241)
0806 04:52:42.803348 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:42.803363 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:42.803393 (+ 30us) server_negotiation.cc:407] Connection header received
0806 04:52:42.803462 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:42.803469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:42.803680 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:42.803851 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:42.805009 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:42.805014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:42.805026 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:52:42.805036 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:42.805074 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:52:42.805087 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:42.805256 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:42.805325 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 04:52:42.805421 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":379,"threads_started":1}
I20260806 04:52:42.806891 24422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:42.899511 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 24296
I20260806 04:52:42.916365 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 23930
I20260806 04:52:42.917650 23993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:43.015353 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 23930
2026-08-06T04:52:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2173 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:52:43 dist-test-slave-1x4g krb5kdc[24445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:52:43 dist-test-slave-1x4g krb5kdc[24445](info): set up 2 sockets
Aug 06 04:52:43 dist-test-slave-1x4g krb5kdc[24445](info): commencing operation
krb5kdc: starting...
W20260806 04:52:44.042326 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:52:44 dist-test-slave-1x4g krb5kdc[24445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:52:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:52:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:52:44.247478 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:34501
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:34507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:34501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:44.544968 24459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:44.545197 24459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:44.545359 24459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:44.551904 24459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:52:44.551994 24459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:44.552037 24459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:44.552110 24459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:52:44.552214 24459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:52:44.560853 24459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:34507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:34501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:34501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:44.561986 24459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:44.563198 24459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:44.573863 24464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:44.573786 24465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:44.573822 24467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:44.574625 24459 server_base.cc:1034] running on GCE node
I20260806 04:52:44.575304 24459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:44.576457 24459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:44.577679 24459 hybrid_clock.cc:648] HybridClock initialized: now 1785991964577623 us; error 43 us; skew 500 ppm
I20260806 04:52:44.578013 24459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:44 dist-test-slave-1x4g krb5kdc[24445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991964, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:44.580925 24459 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:52:44.582264 24459 webserver.cc:460] Webserver started at http://127.13.148.190:41721/ using document root <none> and password file <none>
I20260806 04:52:44.582885 24459 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:44.583014 24459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:44.583336 24459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:44.586179 24459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "dd05a33c9ded41ca994c8f798fc841a0"
format_stamp: "Formatted at 2026-08-06 04:52:44 on dist-test-slave-1x4g"
server_key: "f1bc157c71c9f5837bfd3f34f564284f"
server_key_iv: "b8392b4a32e05b2fa0f2e03e1301a575"
server_key_version: "encryptionkey@0"
I20260806 04:52:44.586836 24459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "dd05a33c9ded41ca994c8f798fc841a0"
format_stamp: "Formatted at 2026-08-06 04:52:44 on dist-test-slave-1x4g"
server_key: "f1bc157c71c9f5837bfd3f34f564284f"
server_key_iv: "b8392b4a32e05b2fa0f2e03e1301a575"
server_key_version: "encryptionkey@0"
I20260806 04:52:44.591439 24459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:52:44.594584 24474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:44.595633 24459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 04:52:44.595798 24459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "dd05a33c9ded41ca994c8f798fc841a0"
format_stamp: "Formatted at 2026-08-06 04:52:44 on dist-test-slave-1x4g"
server_key: "f1bc157c71c9f5837bfd3f34f564284f"
server_key_iv: "b8392b4a32e05b2fa0f2e03e1301a575"
server_key_version: "encryptionkey@0"
I20260806 04:52:44.595978 24459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:44.639365 24459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:44.643340 24459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:44.643635 24459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:44.661514 24459 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:34501
I20260806 04:52:44.661556 24526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:34501 every 8 connection(s)
I20260806 04:52:44.662879 24459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:52:44.665292 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 24459
I20260806 04:52:44.665512 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:52:44.666147 13906 external_mini_cluster.cc:1428] Setting key db963f565be3dfa951d7151edf4e0265
I20260806 04:52:44.666800 24527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:44.680624 24527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0: Bootstrap starting.
Aug 06 04:52:44 dist-test-slave-1x4g krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:52:44.684314 24527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:44.685669 24527 log.cc:824] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0: Log is configured to *not* fsync() on all Append() calls
I20260806 04:52:44.688978 24527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0: No bootstrap required, opened a new log
I20260806 04:52:44.690474 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:44.668160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.13.148.190:34501)
0806 04:52:44.668943 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:44.668957 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:44.669006 (+ 49us) server_negotiation.cc:407] Connection header received
0806 04:52:44.671419 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:44.671454 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:44.671964 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:44.672390 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:44.673881 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:44.675803 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:44.677191 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:44.678290 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:44.678493 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:44.682257 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:44.682267 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:44.682285 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:44.682328 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:44.686104 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:44.686917 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:44.686922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:44.686927 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:44.687135 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:44.687508 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:44.687512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:44.687515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:44.688164 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:44.688476 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:44.688763 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 04:52:44.689199 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":321,"threads_started":1}
I20260806 04:52:44.697033 24527 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd05a33c9ded41ca994c8f798fc841a0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34501 } }
I20260806 04:52:44.697345 24527 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:44.697415 24527 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd05a33c9ded41ca994c8f798fc841a0, State: Initialized, Role: FOLLOWER
I20260806 04:52:44.698004 24527 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd05a33c9ded41ca994c8f798fc841a0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34501 } }
I20260806 04:52:44.698335 24527 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:52:44.698504 24527 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:52:44.698693 24527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:44.701053 24527 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd05a33c9ded41ca994c8f798fc841a0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34501 } }
I20260806 04:52:44.701521 24527 leader_election.cc:302] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd05a33c9ded41ca994c8f798fc841a0; no voters:
I20260806 04:52:44.702092 24527 leader_election.cc:288] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:52:44.702251 24532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:52:44.703003 24532 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [term 1 LEADER]: Becoming Leader. State: Replica: dd05a33c9ded41ca994c8f798fc841a0, State: Running, Role: LEADER
I20260806 04:52:44.703557 24532 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd05a33c9ded41ca994c8f798fc841a0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34501 } }
I20260806 04:52:44.704200 24527 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:52:44.706065 24534 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd05a33c9ded41ca994c8f798fc841a0. Latest consensus state: current_term: 1 leader_uuid: "dd05a33c9ded41ca994c8f798fc841a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd05a33c9ded41ca994c8f798fc841a0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34501 } } }
I20260806 04:52:44.706348 24534 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:44.707168 24537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:52:44.707914 24533 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd05a33c9ded41ca994c8f798fc841a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd05a33c9ded41ca994c8f798fc841a0" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 34501 } } }
I20260806 04:52:44.708261 24533 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0 [sys.catalog]: This master's current role is: LEADER
I20260806 04:52:44.712919 24537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:52:44.721483 24537 catalog_manager.cc:1347] Generated new cluster ID: e1887e585b17483ea1505587f5dd622a
I20260806 04:52:44.721583 24537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:52:44.750216 24537 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:52:44.752071 24537 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:52:44.760525 24537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0: Generated new TSK 0
I20260806 04:52:44.761505 24537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:52:44.818342 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:34501
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:34507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:45.125133 24554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:45.125413 24554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:45.125560 24554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:45.131878 24554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:45.132014 24554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:45.132182 24554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:52:45.141013 24554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:34507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:34501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:45.142084 24554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:45.143311 24554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:45.155550 24559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:45.155632 24562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:45.155591 24560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:45.156531 24554 server_base.cc:1034] running on GCE node
I20260806 04:52:45.157140 24554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:45.157961 24554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:45.159168 24554 hybrid_clock.cc:648] HybridClock initialized: now 1785991965159130 us; error 47 us; skew 500 ppm
I20260806 04:52:45.159483 24554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:45 dist-test-slave-1x4g krb5kdc[24445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991965, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:45.162590 24554 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:52:45.163816 24554 webserver.cc:460] Webserver started at http://127.13.148.129:42095/ using document root <none> and password file <none>
I20260806 04:52:45.164456 24554 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:45.164548 24554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:45.164815 24554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:45.167407 24554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "3c1afb240b3d4a75b3869a65c2b509aa"
format_stamp: "Formatted at 2026-08-06 04:52:45 on dist-test-slave-1x4g"
server_key: "a4e2b2ee8803aaa2d0fc8378634638d6"
server_key_iv: "ad7e7491c126a2c76b0c9cae925947af"
server_key_version: "encryptionkey@0"
I20260806 04:52:45.168092 24554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "3c1afb240b3d4a75b3869a65c2b509aa"
format_stamp: "Formatted at 2026-08-06 04:52:45 on dist-test-slave-1x4g"
server_key: "a4e2b2ee8803aaa2d0fc8378634638d6"
server_key_iv: "ad7e7491c126a2c76b0c9cae925947af"
server_key_version: "encryptionkey@0"
I20260806 04:52:45.173316 24554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260806 04:52:45.176333 24569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:45.177683 24554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:52:45.177824 24554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "3c1afb240b3d4a75b3869a65c2b509aa"
format_stamp: "Formatted at 2026-08-06 04:52:45 on dist-test-slave-1x4g"
server_key: "a4e2b2ee8803aaa2d0fc8378634638d6"
server_key_iv: "ad7e7491c126a2c76b0c9cae925947af"
server_key_version: "encryptionkey@0"
I20260806 04:52:45.177986 24554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:45.190050 24554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:45.194965 24554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:45.195262 24554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:45.196194 24554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:45.197566 24554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:45.197640 24554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:45.197816 24554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:45.197906 24554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:45.227037 24554 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:40193
I20260806 04:52:45.227075 24682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:40193 every 8 connection(s)
I20260806 04:52:45.228253 24554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:52:45.229729 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 24554
I20260806 04:52:45.229926 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:52:45.230731 13906 external_mini_cluster.cc:1428] Setting key 8ec898c4a2298088fad6a952496c12fc
Aug 06 04:52:45 dist-test-slave-1x4g krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991965, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:52:45.247936 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:45.231099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:53297 (local address 127.13.148.190:34501)
0806 04:52:45.231581 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:45.231585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:45.232664 (+ 1079us) server_negotiation.cc:407] Connection header received
0806 04:52:45.234590 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:45.234595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:45.234684 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:45.234794 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:45.236203 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:45.237007 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:45.238554 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:45.238954 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:45.239089 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:45.242314 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:45.242320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:45.242325 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:45.242368 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:45.245049 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:45.245851 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:45.245854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:45.245856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:45.245966 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:45.246440 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:45.246443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:45.246444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:45.246669 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:45.246765 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:45.247565 (+ 800us) server_negotiation.cc:299] Negotiation successful
0806 04:52:45.247703 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":263,"threads_started":1}
I20260806 04:52:45.249459 24686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:45.231725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:34501 (local address 127.13.148.129:53297)
0806 04:52:45.232518 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:45.232570 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:45.234312 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:45.235053 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:45.235061 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:45.235517 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:45.236023 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:45.236040 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:45.237154 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:45.237159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:45.238366 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:45.238377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:45.239179 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:45.239188 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:45.239429 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:45.240080 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:45.240098 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:45.242092 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:45.245187 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:45.245197 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:45.245211 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:45.245756 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:45.246061 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:45.246067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:45.246072 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:45.246330 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:45.246783 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:45.246809 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:45.247209 (+ 400us) client_negotiation.cc:769] Sending connection context
0806 04:52:45.247508 (+ 299us) client_negotiation.cc:240] Negotiation successful
0806 04:52:45.248029 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":290,"threads_started":1}
I20260806 04:52:45.250927 24683 heartbeater.cc:347] Connected to a master server at 127.13.148.190:34501
I20260806 04:52:45.251211 24683 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:45.251859 24683 heartbeater.cc:511] Master 127.13.148.190:34501 requested a full tablet report, sending...
I20260806 04:52:45.253997 24491 ts_manager.cc:194] Registered new tserver with Master: 3c1afb240b3d4a75b3869a65c2b509aa (127.13.148.129:40193)
I20260806 04:52:45.255370 24491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:53297
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:52:45.285943 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:34501
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:34507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:45.587726 24689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:45.587972 24689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:45.588153 24689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:45.594676 24689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:45.594776 24689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:45.594908 24689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:52:45.603528 24689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:34507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:34501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:45.604696 24689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:45.605865 24689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:45.618564 24694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:45.618461 24695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:45.618893 24697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:45.619843 24689 server_base.cc:1034] running on GCE node
I20260806 04:52:45.620394 24689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:45.621317 24689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:45.622534 24689 hybrid_clock.cc:648] HybridClock initialized: now 1785991965622503 us; error 57 us; skew 500 ppm
I20260806 04:52:45.622818 24689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:45 dist-test-slave-1x4g krb5kdc[24445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991965, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:45.628865 24689 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:52:45.630128 24689 webserver.cc:460] Webserver started at http://127.13.148.130:34283/ using document root <none> and password file <none>
I20260806 04:52:45.630764 24689 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:45.630899 24689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:45.631179 24689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:45.634038 24689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "377531e2861f4e8fb9a6698c9ba29a56"
format_stamp: "Formatted at 2026-08-06 04:52:45 on dist-test-slave-1x4g"
server_key: "04b2f2bb51f32521735b4c2545315de1"
server_key_iv: "154bbff3606bd131ec45b0be4f404660"
server_key_version: "encryptionkey@0"
I20260806 04:52:45.634821 24689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "377531e2861f4e8fb9a6698c9ba29a56"
format_stamp: "Formatted at 2026-08-06 04:52:45 on dist-test-slave-1x4g"
server_key: "04b2f2bb51f32521735b4c2545315de1"
server_key_iv: "154bbff3606bd131ec45b0be4f404660"
server_key_version: "encryptionkey@0"
I20260806 04:52:45.639751 24689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 04:52:45.642711 24705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:45.643975 24689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:52:45.644145 24689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "377531e2861f4e8fb9a6698c9ba29a56"
format_stamp: "Formatted at 2026-08-06 04:52:45 on dist-test-slave-1x4g"
server_key: "04b2f2bb51f32521735b4c2545315de1"
server_key_iv: "154bbff3606bd131ec45b0be4f404660"
server_key_version: "encryptionkey@0"
I20260806 04:52:45.644330 24689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:45.656930 24689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:45.660241 24689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:45.660523 24689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:45.661471 24689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:45.662633 24689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:45.662731 24689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:45.662863 24689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:45.662933 24689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:45.695938 24689 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:46231
I20260806 04:52:45.695978 24818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:46231 every 8 connection(s)
I20260806 04:52:45.697335 24689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:52:45.707649 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 24689
I20260806 04:52:45.707805 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:52:45.708257 13906 external_mini_cluster.cc:1428] Setting key 2e98d8917bd90f0b5971660f6f1b77cb
Aug 06 04:52:45 dist-test-slave-1x4g krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991965, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:52:45.716856 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:45.699746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:58861 (local address 127.13.148.190:34501)
0806 04:52:45.699951 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:45.699957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:45.701163 (+ 1206us) server_negotiation.cc:407] Connection header received
0806 04:52:45.702780 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:45.702784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:45.702848 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:45.702931 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:45.704434 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:45.705201 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:45.706630 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:45.706984 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:45.707065 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:45.710598 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:45.710604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:45.710608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:45.710641 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:45.714067 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:45.714772 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:45.714781 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:45.714784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:45.714909 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:45.715301 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:45.715306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:45.715309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:45.715554 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:45.715696 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:45.716285 (+ 589us) server_negotiation.cc:299] Negotiation successful
0806 04:52:45.716498 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 04:52:45.717734 24821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:45.700238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:34501 (local address 127.13.148.130:58861)
0806 04:52:45.701018 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:45.701053 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:45.702560 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:45.703142 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:45.703149 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:45.703649 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:45.704279 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:45.704297 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:45.705332 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:45.705336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:45.706524 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:45.706535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:45.707141 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:45.707145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:45.707306 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:45.707938 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:45.707969 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:45.710398 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:45.714203 (+ 3805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:45.714208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:45.714218 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:45.714644 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:45.715024 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:45.715028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:45.715031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:45.715195 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:45.715672 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:45.715679 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:45.715961 (+ 282us) client_negotiation.cc:769] Sending connection context
0806 04:52:45.716196 (+ 235us) client_negotiation.cc:240] Negotiation successful
0806 04:52:45.716686 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":298,"threads_started":1}
I20260806 04:52:45.718956 24819 heartbeater.cc:347] Connected to a master server at 127.13.148.190:34501
I20260806 04:52:45.719264 24819 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:45.720009 24819 heartbeater.cc:511] Master 127.13.148.190:34501 requested a full tablet report, sending...
I20260806 04:52:45.721349 24491 ts_manager.cc:194] Registered new tserver with Master: 377531e2861f4e8fb9a6698c9ba29a56 (127.13.148.130:46231)
I20260806 04:52:45.722087 24491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:58861
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
I20260806 04:52:45.759925 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:34501
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:34507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:52:46.058951 24824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:52:46.059176 24824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:52:46.059252 24824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:52:46.065629 24824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:52:46.065712 24824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:52:46.065815 24824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:52:46.074050 24824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:34507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:34501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:52:46.075098 24824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:52:46.076175 24824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:52:46.088341 24832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:46.088382 24830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:52:46.088424 24829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:52:46.089424 24824 server_base.cc:1034] running on GCE node
I20260806 04:52:46.089934 24824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:52:46.090695 24824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:52:46.091899 24824 hybrid_clock.cc:648] HybridClock initialized: now 1785991966091860 us; error 51 us; skew 500 ppm
I20260806 04:52:46.092178 24824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:52:46 dist-test-slave-1x4g krb5kdc[24445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991966, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:52:46.095157 24824 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:52:46.096318 24824 webserver.cc:460] Webserver started at http://127.13.148.131:40365/ using document root <none> and password file <none>
I20260806 04:52:46.097035 24824 fs_manager.cc:359] Metadata directory not provided
I20260806 04:52:46.097182 24824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:52:46.097486 24824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:52:46.100163 24824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "81e04c361d0e4e11b23a3cd4f36ef2be"
format_stamp: "Formatted at 2026-08-06 04:52:46 on dist-test-slave-1x4g"
server_key: "64f092cdab80f82d1b0bbdb47d7bf5c8"
server_key_iv: "1361862e170d26fca2402061dd32a0cd"
server_key_version: "encryptionkey@0"
I20260806 04:52:46.100952 24824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "81e04c361d0e4e11b23a3cd4f36ef2be"
format_stamp: "Formatted at 2026-08-06 04:52:46 on dist-test-slave-1x4g"
server_key: "64f092cdab80f82d1b0bbdb47d7bf5c8"
server_key_iv: "1361862e170d26fca2402061dd32a0cd"
server_key_version: "encryptionkey@0"
I20260806 04:52:46.105690 24824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:52:46.108609 24839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:46.109750 24824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:52:46.109923 24824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "81e04c361d0e4e11b23a3cd4f36ef2be"
format_stamp: "Formatted at 2026-08-06 04:52:46 on dist-test-slave-1x4g"
server_key: "64f092cdab80f82d1b0bbdb47d7bf5c8"
server_key_iv: "1361862e170d26fca2402061dd32a0cd"
server_key_version: "encryptionkey@0"
I20260806 04:52:46.110098 24824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:52:46.125263 24824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:52:46.128890 24824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:52:46.129153 24824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:52:46.129905 24824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:52:46.131098 24824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:52:46.131162 24824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:46.131325 24824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:52:46.131371 24824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:52:46.162398 24824 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:37311
I20260806 04:52:46.162461 24952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:37311 every 8 connection(s)
I20260806 04:52:46.163599 24824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:52:46.169350 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 24824
I20260806 04:52:46.169521 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:52:46.169929 13906 external_mini_cluster.cc:1428] Setting key 4edab8e781aad2073121979e5751dfe2
Aug 06 04:52:46 dist-test-slave-1x4g krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991966, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:52:46.182310 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:46.166055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:38569 (local address 127.13.148.190:34501)
0806 04:52:46.166216 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:46.166220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:46.167220 (+ 1000us) server_negotiation.cc:407] Connection header received
0806 04:52:46.168900 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:46.168905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:46.169021 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:46.169124 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:46.170716 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.171758 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.173419 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.173832 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.173970 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:46.177200 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:46.177206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:46.177210 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:46.177250 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:46.179662 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:46.180304 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:46.180314 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:46.180317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:46.180433 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:46.180844 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:46.180851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:46.180855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:46.181104 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:46.181253 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:46.181814 (+ 561us) server_negotiation.cc:299] Negotiation successful
0806 04:52:46.182016 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:52:46.183045 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:46.166475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:34501 (local address 127.13.148.131:38569)
0806 04:52:46.167085 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:46.167117 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:46.168622 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:46.169350 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:46.169357 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:46.169896 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:52:46.170544 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:46.170570 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.171884 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.171889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:46.173281 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:46.173293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.174063 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.174071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:46.174315 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:46.174939 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:52:46.174963 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:52:46.177011 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:52:46.179790 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:46.179798 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:46.179812 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:46.180184 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:46.180530 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:52:46.180534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:52:46.180537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:52:46.180703 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:52:46.181221 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:52:46.181228 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:52:46.181513 (+ 285us) client_negotiation.cc:769] Sending connection context
0806 04:52:46.181735 (+ 222us) client_negotiation.cc:240] Negotiation successful
0806 04:52:46.182170 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":238,"threads_started":1}
I20260806 04:52:46.184397 24953 heartbeater.cc:347] Connected to a master server at 127.13.148.190:34501
I20260806 04:52:46.184741 24953 heartbeater.cc:464] Registering TS with master...
I20260806 04:52:46.185518 24953 heartbeater.cc:511] Master 127.13.148.190:34501 requested a full tablet report, sending...
I20260806 04:52:46.186743 24490 ts_manager.cc:194] Registered new tserver with Master: 81e04c361d0e4e11b23a3cd4f36ef2be (127.13.148.131:37311)
I20260806 04:52:46.187546 24490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:38569
I20260806 04:52:46.197760 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:52:46.213701 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:46.202010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52946 (local address 127.13.148.190:34501)
0806 04:52:46.202207 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:46.202214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:46.202653 (+ 439us) server_negotiation.cc:407] Connection header received
0806 04:52:46.202767 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:46.202771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:46.202823 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:46.202895 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:46.204258 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.205109 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.206208 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.206597 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.206684 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:46.208580 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:46.208585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:46.208587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:46.208613 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:46.211227 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:46.211731 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:46.211734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:46.211736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:46.211834 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:46.212125 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:46.212127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:46.212129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:46.212404 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:46.212568 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:46.212897 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 04:52:46.213049 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:52:46.217661 24490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:52:46.223958 24490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:52:46.247581 24970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:46.238285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54630 (local address 127.13.148.129:40193)
0806 04:52:46.240459 (+ 2174us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:46.240471 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:46.240504 (+ 33us) server_negotiation.cc:407] Connection header received
0806 04:52:46.240634 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:46.240644 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:46.240962 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:46.241260 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:46.241938 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.243077 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.245489 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.246367 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.246589 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:46.246755 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:46.247048 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 04:52:46.247266 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1985,"thread_start_us":2694,"threads_started":1}
I20260806 04:52:46.247622 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:46.238021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:40193 (local address 127.0.0.1:54630)
0806 04:52:46.238552 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:46.238579 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:46.238712 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:46.241288 (+ 2576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:46.241299 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:46.241342 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:46.241748 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:46.241758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.243224 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.243229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:46.245340 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:46.245352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.246588 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.246594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:46.246804 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:46.246839 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 04:52:46.246988 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 04:52:46.247180 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":5,"thread_start_us":257,"threads_started":1}
I20260806 04:52:46.249840 24965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:46.235963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:46231 (local address 127.0.0.1:58762)
0806 04:52:46.237879 (+ 1916us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:46.237898 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:46.238021 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:46.242647 (+ 4626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:46.242651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:46.242675 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:46.242986 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:46.242995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.244944 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.244957 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:46.246933 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:46.246947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.249183 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.249191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:46.249334 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:46.249361 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 04:52:46.249457 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 04:52:46.249568 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1737,"thread_start_us":294,"threads_started":1}
I20260806 04:52:46.250113 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:46.236102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58762 (local address 127.13.148.130:46231)
0806 04:52:46.242014 (+ 5912us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:46.242025 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:46.242057 (+ 32us) server_negotiation.cc:407] Connection header received
0806 04:52:46.242166 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:46.242175 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:46.242431 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:46.242649 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:46.243191 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.244743 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.247110 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.248966 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.249140 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:46.249290 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:46.249500 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 04:52:46.249646 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5713,"thread_start_us":5563,"threads_started":1}
I20260806 04:52:46.251111 24967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:46.235227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:37311 (local address 127.0.0.1:50020)
0806 04:52:46.236736 (+ 1509us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:46.236804 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:46.237017 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:46.244171 (+ 7154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:46.244176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:46.244202 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:46.244654 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:46.244664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.246570 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.246575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:46.248944 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:46.248957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.250465 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.250470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:46.250605 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:46.250630 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 04:52:46.250716 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 04:52:46.250817 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1194,"thread_start_us":2348,"threads_started":1}
I20260806 04:52:46.251178 24966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:46.235708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50020 (local address 127.13.148.131:37311)
0806 04:52:46.243360 (+ 7652us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:46.243365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:46.243390 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:52:46.243494 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:46.243500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:46.243710 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:46.243887 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:46.244830 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.246392 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.249116 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.250304 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.250476 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:46.250624 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:46.250752 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 04:52:46.250878 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7498,"thread_start_us":7338,"threads_started":1}
I20260806 04:52:46.251471 24617 tablet_service.cc:1467] Processing CreateTablet for tablet 7959733cf8e141a4b4d8a1d7c928af34 (DEFAULT_TABLE table=test-table [id=06d9169d950f4a0ca3ee4dbc0aeb7113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:46.253190 24617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7959733cf8e141a4b4d8a1d7c928af34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:46.254655 24753 tablet_service.cc:1467] Processing CreateTablet for tablet 7959733cf8e141a4b4d8a1d7c928af34 (DEFAULT_TABLE table=test-table [id=06d9169d950f4a0ca3ee4dbc0aeb7113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:46.254655 24888 tablet_service.cc:1467] Processing CreateTablet for tablet 7959733cf8e141a4b4d8a1d7c928af34 (DEFAULT_TABLE table=test-table [id=06d9169d950f4a0ca3ee4dbc0aeb7113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:52:46.255932 24888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7959733cf8e141a4b4d8a1d7c928af34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:46.255939 24753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7959733cf8e141a4b4d8a1d7c928af34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:52:46.258349 24683 heartbeater.cc:503] Master 127.13.148.190:34501 was elected leader, sending a full tablet report...
I20260806 04:52:46.264854 24971 tablet_bootstrap.cc:492] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa: Bootstrap starting.
I20260806 04:52:46.268263 24971 tablet_bootstrap.cc:654] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:46.269383 24972 tablet_bootstrap.cc:492] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56: Bootstrap starting.
I20260806 04:52:46.269474 24971 log.cc:824] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa: Log is configured to *not* fsync() on all Append() calls
I20260806 04:52:46.273803 24972 tablet_bootstrap.cc:654] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:46.273813 24973 tablet_bootstrap.cc:492] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be: Bootstrap starting.
I20260806 04:52:46.275518 24971 tablet_bootstrap.cc:492] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa: No bootstrap required, opened a new log
I20260806 04:52:46.275607 24972 log.cc:824] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56: Log is configured to *not* fsync() on all Append() calls
I20260806 04:52:46.275785 24971 ts_tablet_manager.cc:1397] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260806 04:52:46.277002 24973 tablet_bootstrap.cc:654] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be: Neither blocks nor log segments found. Creating new log.
I20260806 04:52:46.277890 24973 log.cc:824] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be: Log is configured to *not* fsync() on all Append() calls
I20260806 04:52:46.278470 24972 tablet_bootstrap.cc:492] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56: No bootstrap required, opened a new log
I20260806 04:52:46.278694 24972 ts_tablet_manager.cc:1397] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260806 04:52:46.280043 24973 tablet_bootstrap.cc:492] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be: No bootstrap required, opened a new log
I20260806 04:52:46.280292 24973 ts_tablet_manager.cc:1397] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260806 04:52:46.285573 24971 raft_consensus.cc:348] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e04c361d0e4e11b23a3cd4f36ef2be" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37311 } } peers { permanent_uuid: "377531e2861f4e8fb9a6698c9ba29a56" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46231 } } peers { permanent_uuid: "3c1afb240b3d4a75b3869a65c2b509aa" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40193 } }
I20260806 04:52:46.285957 24971 raft_consensus.cc:374] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:46.286064 24971 raft_consensus.cc:729] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c1afb240b3d4a75b3869a65c2b509aa, State: Initialized, Role: FOLLOWER
I20260806 04:52:46.286624 24971 consensus_queue.cc:260] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e04c361d0e4e11b23a3cd4f36ef2be" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37311 } } peers { permanent_uuid: "377531e2861f4e8fb9a6698c9ba29a56" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46231 } } peers { permanent_uuid: "3c1afb240b3d4a75b3869a65c2b509aa" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40193 } }
I20260806 04:52:46.288213 24971 ts_tablet_manager.cc:1428] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260806 04:52:46.288125 24972 raft_consensus.cc:348] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e04c361d0e4e11b23a3cd4f36ef2be" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37311 } } peers { permanent_uuid: "377531e2861f4e8fb9a6698c9ba29a56" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46231 } } peers { permanent_uuid: "3c1afb240b3d4a75b3869a65c2b509aa" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40193 } }
I20260806 04:52:46.288519 24972 raft_consensus.cc:374] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:46.288635 24972 raft_consensus.cc:729] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 377531e2861f4e8fb9a6698c9ba29a56, State: Initialized, Role: FOLLOWER
I20260806 04:52:46.288506 24973 raft_consensus.cc:348] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e04c361d0e4e11b23a3cd4f36ef2be" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37311 } } peers { permanent_uuid: "377531e2861f4e8fb9a6698c9ba29a56" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46231 } } peers { permanent_uuid: "3c1afb240b3d4a75b3869a65c2b509aa" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40193 } }
I20260806 04:52:46.288899 24973 raft_consensus.cc:374] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:52:46.289027 24973 raft_consensus.cc:729] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e04c361d0e4e11b23a3cd4f36ef2be, State: Initialized, Role: FOLLOWER
I20260806 04:52:46.289276 24972 consensus_queue.cc:260] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e04c361d0e4e11b23a3cd4f36ef2be" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37311 } } peers { permanent_uuid: "377531e2861f4e8fb9a6698c9ba29a56" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46231 } } peers { permanent_uuid: "3c1afb240b3d4a75b3869a65c2b509aa" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40193 } }
I20260806 04:52:46.289552 24973 consensus_queue.cc:260] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e04c361d0e4e11b23a3cd4f36ef2be" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37311 } } peers { permanent_uuid: "377531e2861f4e8fb9a6698c9ba29a56" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46231 } } peers { permanent_uuid: "3c1afb240b3d4a75b3869a65c2b509aa" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40193 } }
I20260806 04:52:46.290365 24819 heartbeater.cc:503] Master 127.13.148.190:34501 was elected leader, sending a full tablet report...
I20260806 04:52:46.290777 24972 ts_tablet_manager.cc:1428] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260806 04:52:46.292027 24953 heartbeater.cc:503] Master 127.13.148.190:34501 was elected leader, sending a full tablet report...
I20260806 04:52:46.292526 24973 ts_tablet_manager.cc:1428] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260806 04:52:46.295778 24979 raft_consensus.cc:482] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:52:46.296013 24979 raft_consensus.cc:504] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e04c361d0e4e11b23a3cd4f36ef2be" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37311 } } peers { permanent_uuid: "377531e2861f4e8fb9a6698c9ba29a56" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46231 } } peers { permanent_uuid: "3c1afb240b3d4a75b3869a65c2b509aa" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40193 } }
I20260806 04:52:46.297631 24979 leader_election.cc:288] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 377531e2861f4e8fb9a6698c9ba29a56 (127.13.148.130:46231), 3c1afb240b3d4a75b3869a65c2b509aa (127.13.148.129:40193)
I20260806 04:52:46.303421 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:46.297827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:40193 (local address 127.13.148.131:39619)
0806 04:52:46.298079 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:46.298098 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:46.298206 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:46.298634 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:46.298638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:46.298662 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:46.298877 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:46.298888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.300194 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.300198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:46.301803 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:46.301818 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.302886 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.302892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:46.303035 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:46.303057 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:52:46.303127 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 04:52:46.303212 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":50}
I20260806 04:52:46.303562 24970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:46.298132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:39619 (local address 127.13.148.129:40193)
0806 04:52:46.298413 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:46.298417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:46.298430 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:52:46.298475 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:46.298478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:46.298523 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:46.298617 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:46.299026 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.300036 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.301951 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.302744 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.302886 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:46.302957 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:46.303174 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 04:52:46.303325 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260806 04:52:46.304526 24638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7959733cf8e141a4b4d8a1d7c928af34" candidate_uuid: "81e04c361d0e4e11b23a3cd4f36ef2be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c1afb240b3d4a75b3869a65c2b509aa" is_pre_election: true
I20260806 04:52:46.304939 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:46.298079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:49319 (local address 127.13.148.130:46231)
0806 04:52:46.298234 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:46.298242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:46.299540 (+ 1298us) server_negotiation.cc:407] Connection header received
0806 04:52:46.299688 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:46.299692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:46.299749 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:46.299828 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:52:46.300381 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.301108 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.302938 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.304112 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.304237 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:46.304282 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:46.304643 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 04:52:46.304721 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:52:46.305075 24638 raft_consensus.cc:2454] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81e04c361d0e4e11b23a3cd4f36ef2be in term 0.
I20260806 04:52:46.305203 24980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:46.297916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:46231 (local address 127.13.148.131:49319)
0806 04:52:46.299416 (+ 1500us) negotiation.cc:107] Waiting for socket to connect
0806 04:52:46.299428 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:52:46.299561 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:52:46.299846 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:52:46.299849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:52:46.299864 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:52:46.300202 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:46.300212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.301242 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.301247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:46.302760 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:52:46.302771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.304307 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.304313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:52:46.304494 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:46.304531 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 04:52:46.304627 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 04:52:46.304747 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1421,"mutex_wait_us":21,"thread_start_us":1212,"threads_started":1}
I20260806 04:52:46.305675 24841 leader_election.cc:302] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c1afb240b3d4a75b3869a65c2b509aa, 81e04c361d0e4e11b23a3cd4f36ef2be; no voters:
I20260806 04:52:46.305994 24979 raft_consensus.cc:2781] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:52:46.305938 24774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7959733cf8e141a4b4d8a1d7c928af34" candidate_uuid: "81e04c361d0e4e11b23a3cd4f36ef2be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "377531e2861f4e8fb9a6698c9ba29a56" is_pre_election: true
I20260806 04:52:46.306098 24979 raft_consensus.cc:482] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:52:46.306200 24979 raft_consensus.cc:3037] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:46.306383 24774 raft_consensus.cc:2454] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81e04c361d0e4e11b23a3cd4f36ef2be in term 0.
I20260806 04:52:46.308473 24979 raft_consensus.cc:504] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e04c361d0e4e11b23a3cd4f36ef2be" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37311 } } peers { permanent_uuid: "377531e2861f4e8fb9a6698c9ba29a56" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46231 } } peers { permanent_uuid: "3c1afb240b3d4a75b3869a65c2b509aa" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40193 } }
I20260806 04:52:46.309135 24979 leader_election.cc:288] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [CANDIDATE]: Term 1 election: Requested vote from peers 377531e2861f4e8fb9a6698c9ba29a56 (127.13.148.130:46231), 3c1afb240b3d4a75b3869a65c2b509aa (127.13.148.129:40193)
I20260806 04:52:46.309494 24774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7959733cf8e141a4b4d8a1d7c928af34" candidate_uuid: "81e04c361d0e4e11b23a3cd4f36ef2be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "377531e2861f4e8fb9a6698c9ba29a56"
I20260806 04:52:46.309589 24638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7959733cf8e141a4b4d8a1d7c928af34" candidate_uuid: "81e04c361d0e4e11b23a3cd4f36ef2be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c1afb240b3d4a75b3869a65c2b509aa"
I20260806 04:52:46.309674 24774 raft_consensus.cc:3037] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:46.309782 24638 raft_consensus.cc:3037] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:52:46.311847 24638 raft_consensus.cc:2454] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81e04c361d0e4e11b23a3cd4f36ef2be in term 1.
I20260806 04:52:46.311908 24774 raft_consensus.cc:2454] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81e04c361d0e4e11b23a3cd4f36ef2be in term 1.
I20260806 04:52:46.312180 24841 leader_election.cc:302] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c1afb240b3d4a75b3869a65c2b509aa, 81e04c361d0e4e11b23a3cd4f36ef2be; no voters:
I20260806 04:52:46.312516 24979 raft_consensus.cc:2781] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:52:46.312945 24979 raft_consensus.cc:686] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [term 1 LEADER]: Becoming Leader. State: Replica: 81e04c361d0e4e11b23a3cd4f36ef2be, State: Running, Role: LEADER
I20260806 04:52:46.313493 24979 consensus_queue.cc:237] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e04c361d0e4e11b23a3cd4f36ef2be" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37311 } } peers { permanent_uuid: "377531e2861f4e8fb9a6698c9ba29a56" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46231 } } peers { permanent_uuid: "3c1afb240b3d4a75b3869a65c2b509aa" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40193 } }
I20260806 04:52:46.317884 24491 catalog_manager.cc:5495] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be reported cstate change: term changed from 0 to 1, leader changed from <none> to 81e04c361d0e4e11b23a3cd4f36ef2be (127.13.148.131). New cstate: current_term: 1 leader_uuid: "81e04c361d0e4e11b23a3cd4f36ef2be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e04c361d0e4e11b23a3cd4f36ef2be" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "377531e2861f4e8fb9a6698c9ba29a56" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c1afb240b3d4a75b3869a65c2b509aa" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40193 } health_report { overall_health: UNKNOWN } } }
I20260806 04:52:46.345424 24966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:46.340000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50022 (local address 127.13.148.131:37311)
0806 04:52:46.340197 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:46.340202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:46.340219 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:52:46.340413 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:46.340416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:46.340492 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:46.340632 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:52:46.341041 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.341693 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.343151 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.343618 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.343754 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:46.344315 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:52:46.344917 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:52:46.345079 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:46.345124 (+ 45us) server_negotiation.cc:299] Negotiation successful
0806 04:52:46.345198 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 04:52:46.352842 24638 raft_consensus.cc:1261] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa [term 1 FOLLOWER]: Refusing update from remote peer 81e04c361d0e4e11b23a3cd4f36ef2be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:52:46.352921 24774 raft_consensus.cc:1261] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56 [term 1 FOLLOWER]: Refusing update from remote peer 81e04c361d0e4e11b23a3cd4f36ef2be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:52:46.353871 24981 consensus_queue.cc:1035] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [LEADER]: Connected to new peer: Peer: permanent_uuid: "377531e2861f4e8fb9a6698c9ba29a56" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:52:46.354197 24979 consensus_queue.cc:1035] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c1afb240b3d4a75b3869a65c2b509aa" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 40193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:52:46.362401 24984 mvcc.cc:205] Tried to move back new op lower bound from 7315423094172340224 to 7315423094024585216. Current Snapshot: MvccSnapshot[applied={T|T < 7315423094172340224}]
I20260806 04:52:46.365649 24986 mvcc.cc:205] Tried to move back new op lower bound from 7315423094172340224 to 7315423094024585216. Current Snapshot: MvccSnapshot[applied={T|T < 7315423094172340224}]
I20260806 04:52:46.376013 24491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:52:46.376246 24491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:52:46.379509 24491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd05a33c9ded41ca994c8f798fc841a0: Sending DeleteTablet for 3 replicas of tablet 7959733cf8e141a4b4d8a1d7c928af34
I20260806 04:52:46.380453 24888 tablet_service.cc:1514] Processing DeleteTablet for tablet 7959733cf8e141a4b4d8a1d7c928af34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:46 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:50020
I20260806 04:52:46.380534 24753 tablet_service.cc:1514] Processing DeleteTablet for tablet 7959733cf8e141a4b4d8a1d7c928af34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:46 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:58762
I20260806 04:52:46.380715 24617 tablet_service.cc:1514] Processing DeleteTablet for tablet 7959733cf8e141a4b4d8a1d7c928af34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:52:46 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:54630
I20260806 04:52:46.382805 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 24554
I20260806 04:52:46.397274 24994 tablet_replica.cc:330] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be: stopping tablet replica
I20260806 04:52:46.397716 24994 raft_consensus.cc:2229] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:52:46.398358 24994 raft_consensus.cc:2258] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:46.402456 24994 ts_tablet_manager.cc:1905] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:46.405812 24995 tablet_replica.cc:330] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56: stopping tablet replica
I20260806 04:52:46.406388 24995 raft_consensus.cc:2229] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:52:46.407016 24996 tablet_replica.cc:330] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa: stopping tablet replica
I20260806 04:52:46.407469 24996 raft_consensus.cc:2229] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:52:46.407984 24996 raft_consensus.cc:2258] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:46.408689 24995 raft_consensus.cc:2258] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:52:46.412531 24995 ts_tablet_manager.cc:1905] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:46.412573 24994 ts_tablet_manager.cc:1918] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:52:46.412751 24994 log.cc:1196] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/7959733cf8e141a4b4d8a1d7c928af34
I20260806 04:52:46.413309 24994 ts_tablet_manager.cc:1939] T 7959733cf8e141a4b4d8a1d7c928af34 P 81e04c361d0e4e11b23a3cd4f36ef2be: Deleting consensus metadata
I20260806 04:52:46.415084 24476 catalog_manager.cc:4841] TS 81e04c361d0e4e11b23a3cd4f36ef2be (127.13.148.131:37311): tablet 7959733cf8e141a4b4d8a1d7c928af34 (table test-table [id=06d9169d950f4a0ca3ee4dbc0aeb7113]) successfully deleted
Aug 06 04:52:46 dist-test-slave-1x4g krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:52:46.415704 24996 ts_tablet_manager.cc:1905] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:52:46.420935 24996 ts_tablet_manager.cc:1918] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:52:46.421092 24996 log.cc:1196] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/7959733cf8e141a4b4d8a1d7c928af34
I20260806 04:52:46.421550 24996 ts_tablet_manager.cc:1939] T 7959733cf8e141a4b4d8a1d7c928af34 P 3c1afb240b3d4a75b3869a65c2b509aa: Deleting consensus metadata
I20260806 04:52:46.423131 24476 catalog_manager.cc:4841] TS 3c1afb240b3d4a75b3869a65c2b509aa (127.13.148.129:40193): tablet 7959733cf8e141a4b4d8a1d7c928af34 (table test-table [id=06d9169d950f4a0ca3ee4dbc0aeb7113]) successfully deleted
I20260806 04:52:46.424374 24970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:46.383725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.13.148.129:40193)
0806 04:52:46.383890 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:46.383896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:46.404906 (+ 21010us) server_negotiation.cc:407] Connection header received
0806 04:52:46.405123 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:46.405128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:46.405233 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:46.405347 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:46.407278 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.408375 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.410285 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.410894 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.410998 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:46.416933 (+ 5935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:46.416948 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:46.416970 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:46.417024 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:46.420997 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:46.421835 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:46.421844 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:46.421847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:46.421983 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:46.422383 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:46.422387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:46.422390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:46.422878 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:46.423159 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:46.423786 (+ 627us) server_negotiation.cc:299] Negotiation successful
0806 04:52:46.424014 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:52:46.425390 24678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:46.425655 24995 ts_tablet_manager.cc:1918] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:52:46.425801 24995 log.cc:1196] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/7959733cf8e141a4b4d8a1d7c928af34
I20260806 04:52:46.426373 24995 ts_tablet_manager.cc:1939] T 7959733cf8e141a4b4d8a1d7c928af34 P 377531e2861f4e8fb9a6698c9ba29a56: Deleting consensus metadata
I20260806 04:52:46.427700 24475 catalog_manager.cc:4841] TS 377531e2861f4e8fb9a6698c9ba29a56 (127.13.148.130:46231): tablet 7959733cf8e141a4b4d8a1d7c928af34 (table test-table [id=06d9169d950f4a0ca3ee4dbc0aeb7113]) successfully deleted
I20260806 04:52:46.509377 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 24554
I20260806 04:52:46.529202 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 24689
Aug 06 04:52:46 dist-test-slave-1x4g krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:52:46.545241 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:46.530148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.13.148.130:46231)
0806 04:52:46.530371 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:46.530395 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:46.530420 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:52:46.530494 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:46.530502 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:46.530622 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:46.530738 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:46.531889 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.532997 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.534184 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.534833 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.534968 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:46.538797 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:46.538814 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:46.538834 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:46.538881 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:46.542260 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:46.542928 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:46.542937 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:46.542940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:46.543099 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:46.543521 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:46.543527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:46.543530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:46.544044 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:46.544358 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:46.544609 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 04:52:46.544871 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 04:52:46.546106 24814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:46.627385 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 24689
I20260806 04:52:46.650807 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 24824
Aug 06 04:52:46 dist-test-slave-1x4g krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
I20260806 04:52:46.667416 24966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:52:46.651565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50024 (local address 127.13.148.131:37311)
0806 04:52:46.651761 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 04:52:46.651767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:52:46.651786 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:52:46.651938 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:52:46.651943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:52:46.652025 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:52:46.652130 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:52:46.653410 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.654241 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.655472 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:52:46.655979 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:52:46.656101 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:52:46.660396 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:52:46.660408 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:52:46.660425 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:52:46.660457 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:52:46.664084 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:46.665062 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:46.665070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:46.665073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:46.665275 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:52:46.665716 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:52:46.665721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:52:46.665724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:52:46.666113 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:52:46.666435 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0806 04:52:46.666798 (+ 363us) server_negotiation.cc:299] Negotiation successful
0806 04:52:46.667021 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 04:52:46.668134 24948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:46.786599 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 24824
I20260806 04:52:46.804642 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 24459
I20260806 04:52:46.805866 24522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:52:46.915189 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 24459
2026-08-06T04:52:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3901 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:52:47 dist-test-slave-1x4g krb5kdc[25004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:52:47 dist-test-slave-1x4g krb5kdc[25004](info): set up 2 sockets
Aug 06 04:52:47 dist-test-slave-1x4g krb5kdc[25004](info): commencing operation
krb5kdc: starting...
W20260806 04:52:47.941998 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:52:48 dist-test-slave-1x4g krb5kdc[25004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:52:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:52:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.148.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.148.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.148.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/rangeradmin_127.13.148.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.148.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/rangeradmin_127.13.148.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.148.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.148.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.148.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/rangerlookup_127.13.148.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.148.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/rangerlookup_127.13.148.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.148.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.148.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.148.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/HTTP_127.13.148.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.148.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/HTTP_127.13.148.148@KRBTEST.COM.keytab.
I20260806 04:52:48.233930 13906 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task10hk37/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/postgres -l logfile start
2026-08-06 04:52:50.454 UTC [25034] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 04:52:50.454 UTC [25034] LOG: listening on IPv4 address "127.13.148.148", port 59323
2026-08-06 04:52:50.458 UTC [25034] LOG: listening on Unix socket "/tmp/.s.PGSQL.59323"
2026-08-06 04:52:50.478 UTC [25038] LOG: database system was shut down at 2026-08-06 04:52:49 UTC
2026-08-06 04:52:50.481 UTC [25034] LOG: database system is ready to accept connections
I20260806 04:52:51.385939 13906 mini_postgres.cc:96] Postgres bound to 59323
2026-08-06 04:52:51.405 UTC [25046] FATAL: database "slave" does not exist
127.13.148.148:59323 - accepting connections
I20260806 04:52:51.406704 13906 mini_ranger.cc:161] Starting Ranger...
I20260806 04:52:51.433352 13906 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 04:52:51.852342 13906 mini_ranger.cc:87] Created ranger Postgres database
I20260806 04:52:51.852516 13906 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ranger-admin
2026-08-06 04:52:52,126 [I] DB FLAVOR :POSTGRES
2026-08-06 04:52:52,127 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 04:52:52,127 [I] Checking connection..
2026-08-06 04:52:52,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 04:52:52,348 [I] Checking connection passed.
2026-08-06 04:52:52,348 [I] --------- Verifying version history table ---------
2026-08-06 04:52:52,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:52:52,580 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 04:52:52,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:52:52,821 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 04:52:52,821 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 04:52:52,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 04:52:53.055 UTC [25112] WARNING: there is no transaction in progress
2026-08-06 04:52:53,060 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 04:52:53,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:52:53,297 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 04:52:53,297 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 04:52:53,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 04:52:53,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 04:52:53,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1x4g.c.gcp-upstream.internal','N') ;"
2026-08-06 04:52:53,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 04:52:54,185 [I] Table x_portal_user does not exist in database ranger
2026-08-06 04:52:54,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 04:52:54,403 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 04:52:54,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 04:52:54,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 04:52:54,868 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 04:52:54,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 04:52:55.704 UTC [25222] WARNING: there is no transaction in progress
2026-08-06 04:52:55.718 UTC [25222] WARNING: there is no transaction in progress
2026-08-06 04:52:55.732 UTC [25222] WARNING: there is no transaction in progress
2026-08-06 04:52:55.745 UTC [25222] WARNING: there is no transaction in progress
2026-08-06 04:52:55.758 UTC [25222] WARNING: there is no transaction in progress
2026-08-06 04:52:55.775 UTC [25222] WARNING: there is no transaction in progress
2026-08-06 04:52:55.819 UTC [25222] WARNING: there is no transaction in progress
2026-08-06 04:52:55.825 UTC [25222] WARNING: there is no transaction in progress
2026-08-06 04:52:55.838 UTC [25222] WARNING: there is no transaction in progress
2026-08-06 04:52:55.846 UTC [25222] WARNING: there is no transaction in progress
2026-08-06 04:52:55.856 UTC [25222] WARNING: there is no transaction in progress
2026-08-06 04:52:55.867 UTC [25222] WARNING: there is no transaction in progress
2026-08-06 04:52:55.875 UTC [25222] WARNING: there is no transaction in progress
2026-08-06 04:52:55.881 UTC [25222] WARNING: there is no transaction in progress
2026-08-06 04:52:55.886 UTC [25222] WARNING: there is no transaction in progress
2026-08-06 04:52:55.892 UTC [25222] WARNING: there is no transaction in progress
2026-08-06 04:52:56,412 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 04:52:56,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 04:52:56,615 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 04:52:56,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 04:52:56,835 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 04:52:56,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 04:52:57,069 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 04:52:57,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 04:52:57,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 04:52:57,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1x4g.c.gcp-upstream.internal';"
2026-08-06 04:52:57,714 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 04:52:57,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 04:52:57,924 [I] DB_PATCHES have already been applied
I20260806 04:52:57.930372 13906 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ranger-admin:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task10hk37/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 04:52:57.930481 13906 mini_ranger.cc:193] Using host: 127.13.148.148
I20260806 04:52:57.947674 13906 mini_ranger.cc:227] Ranger admin URL: http://127.13.148.148:41437
Aug 06, 2026 4:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 4:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 4:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 4:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 4:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 04:52:58.983803 13906 mini_ranger.cc:160] Time spent starting Ranger: real 7.577s user 0.000s sys 0.045s
Aug 06, 2026 4:53:16 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 4:53:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 06, 2026 4:53:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 4:53:16 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 4:53:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 4:53:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 4:53:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 4:53:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 4:53:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 4:53:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 4:53:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 4:53:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 4:53:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 4:53:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 4:53:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 4:53:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 4:53:16 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 4:53:17 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 04:53:18.212373 13906 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.148.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.148.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.148.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/rangerkms_127.13.148.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.148.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/rangerkms_127.13.148.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.148.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.148.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.148.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/HTTP_127.13.148.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.148.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/HTTP_127.13.148.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 04:53:18 dist-test-slave-1x4g krb5kdc[25004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991998, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 04:53:18.385295 13906 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 04:53:18.405444 13906 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 04:53:19.130463 13906 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 04:53:19.130599 13906 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ranger-kms
I20260806 04:53:19.130754 13906 mini_ranger_kms.cc:227] Using postgres at 127.13.148.148:59323
2026-08-06 04:53:19,780 [I] DB FLAVOR :POSTGRES
2026-08-06 04:53:19,780 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 04:53:19,780 [I] Checking connection
2026-08-06 04:53:19,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 04:53:20,010 [I] connection success
2026-08-06 04:53:20,010 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 04:53:20,010 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 04:53:20,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 04:53:20,231 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 04:53:20,231 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 04:53:20,231 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 04:53:20,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:59323/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 04:53:20,469 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 04:53:20.895056 13906 mini_ranger_kms.cc:310] Created kms service
I20260806 04:53:21.075026 13906 mini_ranger_kms.cc:326] Created kudu user
I20260806 04:53:21.116658 13906 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 04:53:21 dist-test-slave-1x4g krb5kdc[25004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991998, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
I20260806 04:53:21.482977 13906 mini_ranger_kms.cc:384] Added ranger policy
I20260806 04:53:21.483306 13906 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ranger-kms:/tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 04:53:21.483376 13906 mini_ranger_kms.cc:240] Using host: 127.13.148.148
I20260806 04:53:21.506480 13906 mini_ranger_kms.cc:276] Ranger KMS PID: 25406
I20260806 04:53:21.506675 13906 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.148.148:40725
Aug 06, 2026 4:53:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:53:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:53:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 4:53:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 4:53:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 4:53:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 04:53:23 dist-test-slave-1x4g krb5kdc[25004](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785992003, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:53:23.661911 13906 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.277s user 0.130s sys 0.385s
I20260806 04:53:23.662142 13906 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 04:53:23.662377 13906 mini_ranger_kms.cc:401] 127.13.148.148:40725/kms/v1/keys
Aug 06 04:53:24 dist-test-slave-1x4g krb5kdc[25004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992003, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.148@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
Aug 06 04:53:24 dist-test-slave-1x4g krb5kdc[25004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992003, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.148@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
Aug 06 04:53:26 dist-test-slave-1x4g krb5kdc[25004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:53:26.494287 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:42373
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:44341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:42373
--ranger_config_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.148:40725/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:53:26.813966 25458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:53:26.814213 25458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:53:26.814292 25458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:53:26.820621 25458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:53:26.820704 25458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:53:26.820744 25458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:53:26.820824 25458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:53:26.820892 25458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:53:26.831197 25458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:44341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.148.148:40725/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:42373
--ranger_config_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:42373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:53:26.832644 25458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:53:26.834260 25458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:53:26.846622 25463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:53:26.846691 25466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:53:26.846668 25464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:53:26.846930 25458 server_base.cc:1034] running on GCE node
I20260806 04:53:26.848061 25458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:53:26.849356 25458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:53:26.850548 25458 hybrid_clock.cc:648] HybridClock initialized: now 1785992006850510 us; error 26 us; skew 500 ppm
I20260806 04:53:26.850812 25458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:53:26 dist-test-slave-1x4g krb5kdc[25004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992006, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:53:26.853717 25458 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:53:26.855027 25458 webserver.cc:460] Webserver started at http://127.13.148.190:33867/ using document root <none> and password file <none>
I20260806 04:53:26.855566 25458 fs_manager.cc:359] Metadata directory not provided
I20260806 04:53:26.855647 25458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:53:26.855860 25458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:53:26 dist-test-slave-1x4g krb5kdc[25004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992006, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
I20260806 04:53:26.912201 25458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "d7c41fb34c4f48e185876476856aea50"
format_stamp: "Formatted at 2026-08-06 04:53:26 on dist-test-slave-1x4g"
server_key: "e6e27a75b41b45378946a0c9fca1e600"
server_key_iv: "d259b4439fd54e80e020eba847701bce"
server_key_version: "kuduclusterkey@0"
I20260806 04:53:26.913251 25458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "d7c41fb34c4f48e185876476856aea50"
format_stamp: "Formatted at 2026-08-06 04:53:26 on dist-test-slave-1x4g"
server_key: "e6e27a75b41b45378946a0c9fca1e600"
server_key_iv: "d259b4439fd54e80e020eba847701bce"
server_key_version: "kuduclusterkey@0"
I20260806 04:53:26.920270 25458 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260806 04:53:26.935783 25473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:53:26.937407 25458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:53:26.937599 25458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "d7c41fb34c4f48e185876476856aea50"
format_stamp: "Formatted at 2026-08-06 04:53:26 on dist-test-slave-1x4g"
server_key: "e6e27a75b41b45378946a0c9fca1e600"
server_key_iv: "d259b4439fd54e80e020eba847701bce"
server_key_version: "kuduclusterkey@0"
I20260806 04:53:26.937856 25458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:53:26.956858 25458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:53:26.960562 25458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:53:26.960872 25458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:53:26.984438 25458 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:42373
I20260806 04:53:26.984467 25525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:42373 every 8 connection(s)
I20260806 04:53:26.985860 25458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:53:26.987617 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 25458
I20260806 04:53:26.987830 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:53:26.988642 13906 external_mini_cluster.cc:1428] Setting key ccc8505f9e316f1da36c8ae3d68bcc2a
I20260806 04:53:26.989935 25526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:53:27.004966 25526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50: Bootstrap starting.
I20260806 04:53:27.007768 25526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50: Neither blocks nor log segments found. Creating new log.
I20260806 04:53:27.008714 25526 log.cc:824] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50: Log is configured to *not* fsync() on all Append() calls
I20260806 04:53:27.011384 25526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50: No bootstrap required, opened a new log
Aug 06 04:53:27 dist-test-slave-1x4g krb5kdc[25004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:53:27.019381 25526 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c41fb34c4f48e185876476856aea50" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42373 } }
I20260806 04:53:27.019728 25526 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:53:27.019850 25526 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c41fb34c4f48e185876476856aea50, State: Initialized, Role: FOLLOWER
I20260806 04:53:27.020460 25526 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c41fb34c4f48e185876476856aea50" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42373 } }
I20260806 04:53:27.020812 25526 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:53:27.021023 25526 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:53:27.021242 25526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:53:27.023690 25526 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c41fb34c4f48e185876476856aea50" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42373 } }
I20260806 04:53:27.024178 25526 leader_election.cc:302] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7c41fb34c4f48e185876476856aea50; no voters:
I20260806 04:53:27.024893 25526 leader_election.cc:288] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:53:27.025004 25531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:53:27.025074 25529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:26.990753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.13.148.190:42373)
0806 04:53:27.002102 (+ 11349us) server_negotiation.cc:206] Beginning negotiation
0806 04:53:27.002112 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:53:27.002149 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:53:27.004279 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:53:27.004306 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:53:27.004752 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:53:27.006698 (+ 1946us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:53:27.007204 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:27.009146 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:27.010977 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:27.012035 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:27.012216 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:27.016694 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:53:27.016704 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:53:27.016722 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:53:27.016768 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:53:27.020524 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:53:27.021327 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:53:27.021335 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:53:27.021343 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:53:27.021556 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:53:27.021996 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:53:27.022001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:53:27.022004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:53:27.022665 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:53:27.022947 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0806 04:53:27.023347 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 04:53:27.023717 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11136,"thread_start_us":10927,"threads_started":1}
I20260806 04:53:27.030938 25526 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:53:27.037066 25531 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [term 1 LEADER]: Becoming Leader. State: Replica: d7c41fb34c4f48e185876476856aea50, State: Running, Role: LEADER
I20260806 04:53:27.037814 25531 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c41fb34c4f48e185876476856aea50" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42373 } }
I20260806 04:53:27.049477 25532 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7c41fb34c4f48e185876476856aea50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c41fb34c4f48e185876476856aea50" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42373 } } }
I20260806 04:53:27.049903 25532 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [sys.catalog]: This master's current role is: LEADER
I20260806 04:53:27.051162 25543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:53:27.050952 25533 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7c41fb34c4f48e185876476856aea50. Latest consensus state: current_term: 1 leader_uuid: "d7c41fb34c4f48e185876476856aea50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c41fb34c4f48e185876476856aea50" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42373 } } }
I20260806 04:53:27.051330 25533 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50 [sys.catalog]: This master's current role is: LEADER
I20260806 04:53:27.067602 25543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:53:27.068367 25526 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 04:53:27.093284 25543 catalog_manager.cc:1347] Generated new cluster ID: 2b11cace259349e28a7a5240f93e757d
I20260806 04:53:27.093492 25543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:53:27.121786 25543 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:53:27.123759 25543 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:53:27.136966 25543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50: Generated new TSK 0
I20260806 04:53:27.138342 25543 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:53:27.888473 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.148:40725/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42373
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:44341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:53:28 dist-test-slave-1x4g krb5kdc[25004](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785992008, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 04:53:28.309942 25575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:53:28.310277 25575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:53:28.310416 25575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:53:28.320465 25575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:53:28.320602 25575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:53:28.320828 25575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:53:28.332828 25575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:44341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.148.148:40725/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:53:28.334486 25575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:53:28.336232 25575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:53:28.352227 25586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:53:28.353866 25588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:53:28.354321 25585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:53:28.361958 25575 server_base.cc:1034] running on GCE node
I20260806 04:53:28.362653 25575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:53:28.363822 25575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:53:28.365119 25575 hybrid_clock.cc:648] HybridClock initialized: now 1785992008364972 us; error 66 us; skew 500 ppm
I20260806 04:53:28.365609 25575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:53:28 dist-test-slave-1x4g krb5kdc[25004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992008, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:53:28.374209 25575 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:53:28.391784 25575 webserver.cc:460] Webserver started at http://127.13.148.129:43521/ using document root <none> and password file <none>
I20260806 04:53:28.392520 25575 fs_manager.cc:359] Metadata directory not provided
I20260806 04:53:28.392659 25575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:53:28.393014 25575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:53:28 dist-test-slave-1x4g krb5kdc[25004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992008, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
I20260806 04:53:28.422807 25575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "0d2bae01a25e4186a159900cfca97e77"
format_stamp: "Formatted at 2026-08-06 04:53:28 on dist-test-slave-1x4g"
server_key: "1f0ad3e8e76f9fdc2ba5082dabe6e221"
server_key_iv: "614ee90dcb925a66512d089040f9178d"
server_key_version: "kuduclusterkey@0"
I20260806 04:53:28.423990 25575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "0d2bae01a25e4186a159900cfca97e77"
format_stamp: "Formatted at 2026-08-06 04:53:28 on dist-test-slave-1x4g"
server_key: "1f0ad3e8e76f9fdc2ba5082dabe6e221"
server_key_iv: "614ee90dcb925a66512d089040f9178d"
server_key_version: "kuduclusterkey@0"
I20260806 04:53:28.437561 25575 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.010s sys 0.005s
I20260806 04:53:28.470574 25595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:53:28.481616 25575 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.005s sys 0.008s
I20260806 04:53:28.481837 25575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "0d2bae01a25e4186a159900cfca97e77"
format_stamp: "Formatted at 2026-08-06 04:53:28 on dist-test-slave-1x4g"
server_key: "1f0ad3e8e76f9fdc2ba5082dabe6e221"
server_key_iv: "614ee90dcb925a66512d089040f9178d"
server_key_version: "kuduclusterkey@0"
I20260806 04:53:28.482112 25575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:53:28.514716 25575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:53:28.519825 25575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:53:28.520148 25575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:53:28.521430 25575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:53:28.523212 25575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:53:28.523345 25575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:53:28.523495 25575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:53:28.523569 25575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:53:28 dist-test-slave-1x4g krb5kdc[25004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992008, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
I20260806 04:53:28.698189 25575 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:37319
I20260806 04:53:28.698285 25710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:37319 every 8 connection(s)
I20260806 04:53:28.699534 25575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:53:28.703392 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 25575
I20260806 04:53:28.703586 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:53:28.704295 13906 external_mini_cluster.cc:1428] Setting key 3520f9c2cd45b5f6018f220781ccc80b
Aug 06 04:53:28 dist-test-slave-1x4g krb5kdc[25004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992008, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:53:28.726109 25713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:28.702305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:35965 (local address 127.13.148.190:42373)
0806 04:53:28.702818 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0806 04:53:28.702824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:53:28.703789 (+ 965us) server_negotiation.cc:407] Connection header received
0806 04:53:28.705935 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:53:28.705942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:53:28.706053 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:53:28.706184 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:53:28.707762 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:28.708466 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:28.710045 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:28.710441 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:28.710572 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:28.717659 (+ 7087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:53:28.717667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:53:28.717671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:53:28.717714 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:53:28.721363 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:53:28.722262 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:53:28.722269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:53:28.722275 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:53:28.722421 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:53:28.722908 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:53:28.722914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:53:28.722919 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:53:28.723296 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:53:28.723431 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 04:53:28.724416 (+ 985us) server_negotiation.cc:299] Negotiation successful
0806 04:53:28.725639 (+ 1223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":237,"threads_started":1}
I20260806 04:53:28.728574 25714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:28.702854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42373 (local address 127.13.148.129:35965)
0806 04:53:28.703630 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0806 04:53:28.703681 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 04:53:28.705626 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:53:28.706498 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:53:28.706509 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:53:28.707004 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:53:28.707594 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:53:28.707611 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:53:28.708627 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:53:28.708630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:53:28.709884 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:53:28.709911 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:53:28.710676 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:53:28.710684 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:53:28.710911 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:28.711570 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:53:28.711590 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:53:28.717336 (+ 5746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:53:28.721543 (+ 4207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:53:28.721555 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:53:28.721575 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:53:28.722144 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:53:28.722548 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:53:28.722554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:53:28.722559 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:53:28.722790 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:53:28.723526 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:53:28.723535 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:53:28.723990 (+ 455us) client_negotiation.cc:769] Sending connection context
0806 04:53:28.726447 (+ 2457us) client_negotiation.cc:240] Negotiation successful
0806 04:53:28.727087 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":315,"threads_started":1}
I20260806 04:53:28.730356 25711 heartbeater.cc:347] Connected to a master server at 127.13.148.190:42373
I20260806 04:53:28.730675 25711 heartbeater.cc:464] Registering TS with master...
I20260806 04:53:28.731544 25711 heartbeater.cc:511] Master 127.13.148.190:42373 requested a full tablet report, sending...
I20260806 04:53:28.734120 25491 ts_manager.cc:194] Registered new tserver with Master: 0d2bae01a25e4186a159900cfca97e77 (127.13.148.129:37319)
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
I20260806 04:53:28.735966 25491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:35965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:53:28.765805 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.148:40725/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42373
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:44341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:53:28 dist-test-slave-1x4g krb5kdc[25004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992008, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
W20260806 04:53:29.183575 25717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:53:29.183892 25717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:53:29.184021 25717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:53:29.193321 25717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:53:29.193460 25717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:53:29.193627 25717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:53:29.206808 25717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:44341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.148.148:40725/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:53:29.208343 25717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:53:29.209896 25717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:53:29.225065 25731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:53:29.225524 25732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:53:29.233683 25717 server_base.cc:1034] running on GCE node
W20260806 04:53:29.234016 25734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:53:29.234771 25717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:53:29.236071 25717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:53:29.237370 25717 hybrid_clock.cc:648] HybridClock initialized: now 1785992009237223 us; error 47 us; skew 500 ppm
I20260806 04:53:29.237761 25717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:53:29 dist-test-slave-1x4g krb5kdc[25004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992009, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:53:29.241961 25717 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:53:29.243736 25717 webserver.cc:460] Webserver started at http://127.13.148.130:45573/ using document root <none> and password file <none>
I20260806 04:53:29.244597 25717 fs_manager.cc:359] Metadata directory not provided
I20260806 04:53:29.244729 25717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:53:29.245076 25717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:53:29 dist-test-slave-1x4g krb5kdc[25004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992009, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
I20260806 04:53:29.264458 25717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "f04a18364c4a476e958e613925f4ba2b"
format_stamp: "Formatted at 2026-08-06 04:53:29 on dist-test-slave-1x4g"
server_key: "326de437c02d532d6eb0d2dbec4c1193"
server_key_iv: "42af57bedbe10d8d88e216611217eea5"
server_key_version: "kuduclusterkey@0"
I20260806 04:53:29.265375 25717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "f04a18364c4a476e958e613925f4ba2b"
format_stamp: "Formatted at 2026-08-06 04:53:29 on dist-test-slave-1x4g"
server_key: "326de437c02d532d6eb0d2dbec4c1193"
server_key_iv: "42af57bedbe10d8d88e216611217eea5"
server_key_version: "kuduclusterkey@0"
I20260806 04:53:29.271708 25717 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 04:53:29.290158 25741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:53:29.291874 25717 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:53:29.292044 25717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "f04a18364c4a476e958e613925f4ba2b"
format_stamp: "Formatted at 2026-08-06 04:53:29 on dist-test-slave-1x4g"
server_key: "326de437c02d532d6eb0d2dbec4c1193"
server_key_iv: "42af57bedbe10d8d88e216611217eea5"
server_key_version: "kuduclusterkey@0"
I20260806 04:53:29.292279 25717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:53:29.341902 25717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:53:29.346933 25717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:53:29.347272 25717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:53:29.348595 25717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:53:29.350235 25717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:53:29.350330 25717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:53:29.350435 25717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:53:29.350485 25717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:53:29.413496 25717 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:39713
I20260806 04:53:29.413517 25854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:39713 every 8 connection(s)
I20260806 04:53:29.415050 25717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:53:29.415447 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 25717
I20260806 04:53:29.415582 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:53:29.416038 13906 external_mini_cluster.cc:1428] Setting key 1847ce1dea077907449af8f1c6663bb9
Aug 06 04:53:29 dist-test-slave-1x4g krb5kdc[25004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992009, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:53:29.481530 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:29.425843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:37133 (local address 127.13.148.190:42373)
0806 04:53:29.451771 (+ 25928us) server_negotiation.cc:206] Beginning negotiation
0806 04:53:29.451778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:53:29.455806 (+ 4028us) server_negotiation.cc:407] Connection header received
0806 04:53:29.455877 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:53:29.455882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:53:29.455958 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:53:29.456052 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:53:29.465491 (+ 9439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:29.466698 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:29.469142 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:29.469748 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:29.469846 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:29.474325 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:53:29.474333 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:53:29.474338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:53:29.474393 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:53:29.477942 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:53:29.478776 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:53:29.478782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:53:29.478785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:53:29.478942 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:53:29.479431 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:53:29.479437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:53:29.479440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:53:29.479759 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:53:29.479897 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 04:53:29.480879 (+ 982us) server_negotiation.cc:299] Negotiation successful
0806 04:53:29.481180 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25766,"thread_start_us":25913,"threads_started":1}
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
I20260806 04:53:29.483562 25857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:29.425267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42373 (local address 127.13.148.130:37133)
0806 04:53:29.452557 (+ 27290us) negotiation.cc:107] Waiting for socket to connect
0806 04:53:29.452662 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0806 04:53:29.455318 (+ 2656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:53:29.457271 (+ 1953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:53:29.457284 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:53:29.458070 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:53:29.458947 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:53:29.458971 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:53:29.466937 (+ 7966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:53:29.466944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:53:29.468985 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:53:29.468999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:53:29.470186 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:53:29.470194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:53:29.470474 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:29.471212 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:53:29.471250 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:53:29.474001 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:53:29.478084 (+ 4083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:53:29.478093 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:53:29.478110 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:53:29.478637 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:53:29.479087 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:53:29.479091 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:53:29.479095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:53:29.479313 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:53:29.479983 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:53:29.479991 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:53:29.480413 (+ 422us) client_negotiation.cc:769] Sending connection context
0806 04:53:29.481756 (+ 1343us) client_negotiation.cc:240] Negotiation successful
0806 04:53:29.482255 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26942,"thread_start_us":30754,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
I20260806 04:53:29.485224 25855 heartbeater.cc:347] Connected to a master server at 127.13.148.190:42373
I20260806 04:53:29.485541 25855 heartbeater.cc:464] Registering TS with master...
I20260806 04:53:29.486316 25855 heartbeater.cc:511] Master 127.13.148.190:42373 requested a full tablet report, sending...
I20260806 04:53:29.488008 25491 ts_manager.cc:194] Registered new tserver with Master: f04a18364c4a476e958e613925f4ba2b (127.13.148.130:39713)
I20260806 04:53:29.489018 25491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:37133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
W20260806 04:53:29.519870 13906 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.13.148.131: real 0.101s user 0.001s sys 0.019s
I20260806 04:53:29.520846 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.148:40725/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42373
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:44341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:53:29.742686 25711 heartbeater.cc:503] Master 127.13.148.190:42373 was elected leader, sending a full tablet report...
W20260806 04:53:29.859874 25861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:53:29.860157 25861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:53:29.860250 25861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:53:29.866660 25861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:53:29.866766 25861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:53:29.866990 25861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:53:29.876214 25861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:44341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.148.148:40725/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:42373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:53:29.877573 25861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:53:29.878860 25861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:53:29.891824 25867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:53:29.891887 25866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:53:29.891824 25869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:53:29.892643 25861 server_base.cc:1034] running on GCE node
I20260806 04:53:29.893217 25861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:53:29.894024 25861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:53:29.895208 25861 hybrid_clock.cc:648] HybridClock initialized: now 1785992009895178 us; error 33 us; skew 500 ppm
I20260806 04:53:29.895491 25861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:53:29 dist-test-slave-1x4g krb5kdc[25004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992009, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:53:29.898703 25861 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:53:29.900022 25861 webserver.cc:460] Webserver started at http://127.13.148.131:42729/ using document root <none> and password file <none>
I20260806 04:53:29.900609 25861 fs_manager.cc:359] Metadata directory not provided
I20260806 04:53:29.900712 25861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:53:29.901014 25861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:53:29 dist-test-slave-1x4g krb5kdc[25004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992009, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
I20260806 04:53:29.918227 25861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "594d369b58964892a79392adede5fb92"
format_stamp: "Formatted at 2026-08-06 04:53:29 on dist-test-slave-1x4g"
server_key: "58aed56bb79d86f0dae3411bd77bc266"
server_key_iv: "0b8da7e6549dfc19a492f477be1ba31b"
server_key_version: "kuduclusterkey@0"
I20260806 04:53:29.919206 25861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "594d369b58964892a79392adede5fb92"
format_stamp: "Formatted at 2026-08-06 04:53:29 on dist-test-slave-1x4g"
server_key: "58aed56bb79d86f0dae3411bd77bc266"
server_key_iv: "0b8da7e6549dfc19a492f477be1ba31b"
server_key_version: "kuduclusterkey@0"
I20260806 04:53:29.925552 25861 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 04:53:29.939484 25876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:53:29.940994 25861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:53:29.941148 25861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "594d369b58964892a79392adede5fb92"
format_stamp: "Formatted at 2026-08-06 04:53:29 on dist-test-slave-1x4g"
server_key: "58aed56bb79d86f0dae3411bd77bc266"
server_key_iv: "0b8da7e6549dfc19a492f477be1ba31b"
server_key_version: "kuduclusterkey@0"
I20260806 04:53:29.941447 25861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:53:29.965045 25861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:53:29.968464 25861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:53:29.968737 25861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:53:29.969662 25861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:53:29.970904 25861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:53:29.971024 25861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:53:29.971167 25861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:53:29.971238 25861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:53:30.005416 25861 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:35879
I20260806 04:53:30.005486 25990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:35879 every 8 connection(s)
I20260806 04:53:30.006798 25861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:53:30.009327 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 25861
I20260806 04:53:30.009511 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:53:30.010087 13906 external_mini_cluster.cc:1428] Setting key 7284ff419db7acdaf0c96b31fd51e84c
Aug 06 04:53:30 dist-test-slave-1x4g krb5kdc[25004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992009, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:53:30.027168 25993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:30.009443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:53351 (local address 127.13.148.190:42373)
0806 04:53:30.011304 (+ 1861us) server_negotiation.cc:206] Beginning negotiation
0806 04:53:30.011310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:53:30.011748 (+ 438us) server_negotiation.cc:407] Connection header received
0806 04:53:30.014044 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:53:30.014048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:53:30.014110 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:53:30.014196 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:53:30.015531 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.016298 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.017884 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.018278 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.018367 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:30.021426 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:53:30.021430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:53:30.021433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:53:30.021464 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:53:30.024121 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:53:30.025001 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:53:30.025005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:53:30.025009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:53:30.025154 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:53:30.025653 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:53:30.025659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:53:30.025662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:53:30.026025 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:53:30.026143 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 04:53:30.026757 (+ 614us) server_negotiation.cc:299] Negotiation successful
0806 04:53:30.026908 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1718,"thread_start_us":1540,"threads_started":1}
I20260806 04:53:30.028182 25994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:30.010132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42373 (local address 127.13.148.131:53351)
0806 04:53:30.011545 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0806 04:53:30.011617 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 04:53:30.013801 (+ 2184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:53:30.014401 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:53:30.014409 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:53:30.014825 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:53:30.015387 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:53:30.015409 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.016417 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.016421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:53:30.017739 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:53:30.017771 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.018450 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.018454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:53:30.018639 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:30.019223 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:53:30.019244 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:53:30.021249 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:53:30.024253 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:53:30.024259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:53:30.024270 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:53:30.024878 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:53:30.025274 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:53:30.025277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:53:30.025279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:53:30.025522 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:53:30.026136 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:53:30.026142 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:53:30.026407 (+ 265us) client_negotiation.cc:769] Sending connection context
0806 04:53:30.026654 (+ 247us) client_negotiation.cc:240] Negotiation successful
0806 04:53:30.027001 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"thread_start_us":433,"threads_started":1}
I20260806 04:53:30.029527 25991 heartbeater.cc:347] Connected to a master server at 127.13.148.190:42373
I20260806 04:53:30.029790 25991 heartbeater.cc:464] Registering TS with master...
I20260806 04:53:30.030499 25991 heartbeater.cc:511] Master 127.13.148.190:42373 requested a full tablet report, sending...
I20260806 04:53:30.031891 25491 ts_manager.cc:194] Registered new tserver with Master: 594d369b58964892a79392adede5fb92 (127.13.148.131:35879)
I20260806 04:53:30.032689 25491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:53351
I20260806 04:53:30.037781 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:53:30.054277 25993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:30.042458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.13.148.190:42373)
0806 04:53:30.042672 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 04:53:30.042678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:53:30.042920 (+ 242us) server_negotiation.cc:407] Connection header received
0806 04:53:30.043055 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:53:30.043058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:53:30.043125 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:53:30.043199 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:53:30.044667 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.045400 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.046809 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.047194 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.047285 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:30.049733 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:53:30.049738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:53:30.049741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:53:30.049767 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:53:30.052115 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:53:30.052722 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:53:30.052726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:53:30.052728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:53:30.052856 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:53:30.053162 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:53:30.053165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:53:30.053167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:53:30.053395 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:53:30.053570 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 04:53:30.053793 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 04:53:30.053935 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 04:53:30.058145 25491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:53:30.065090 25491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:53:30.087670 26005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:30.076821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52964 (local address 127.13.148.130:39713)
0806 04:53:30.078494 (+ 1673us) server_negotiation.cc:206] Beginning negotiation
0806 04:53:30.078502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:53:30.078534 (+ 32us) server_negotiation.cc:407] Connection header received
0806 04:53:30.078652 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:53:30.078662 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:53:30.078974 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:53:30.080073 (+ 1099us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:53:30.080316 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.081858 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.084993 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.086403 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.086609 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:30.086772 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 04:53:30.087121 (+ 349us) server_negotiation.cc:299] Negotiation successful
0806 04:53:30.087337 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1497,"thread_start_us":1395,"threads_started":1}
I20260806 04:53:30.087738 26004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:30.076608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:39713 (local address 127.0.0.1:52964)
0806 04:53:30.077354 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0806 04:53:30.077384 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 04:53:30.077526 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:53:30.079263 (+ 1737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:53:30.079270 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:53:30.079305 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:53:30.079834 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:53:30.079848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.082012 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.082020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:53:30.084742 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:53:30.084757 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.086645 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.086653 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:53:30.086842 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:30.086893 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 04:53:30.087062 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 04:53:30.087256 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":372,"threads_started":1}
I20260806 04:53:30.088162 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:30.078955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:37319 (local address 127.0.0.1:47620)
0806 04:53:30.079532 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0806 04:53:30.079556 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 04:53:30.079672 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:53:30.080690 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:53:30.080694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:53:30.080719 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:53:30.081133 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:53:30.081147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.083089 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.083094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:53:30.085196 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:53:30.085206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.087360 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.087368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:53:30.087568 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:30.087598 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 04:53:30.087669 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 04:53:30.087813 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":5,"thread_start_us":645,"threads_started":1}
I20260806 04:53:30.089172 26009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:30.079434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47620 (local address 127.13.148.129:37319)
0806 04:53:30.080256 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0806 04:53:30.080262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:53:30.080284 (+ 22us) server_negotiation.cc:407] Connection header received
0806 04:53:30.080359 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:53:30.080363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:53:30.080512 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:53:30.080709 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:53:30.081336 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.082808 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.085958 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.087136 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.088458 (+ 1322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:30.088642 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 04:53:30.088710 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 04:53:30.088888 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":515,"threads_started":1}
I20260806 04:53:30.089732 26006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:30.077976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:35879 (local address 127.0.0.1:44694)
0806 04:53:30.078882 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0806 04:53:30.078899 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:53:30.079020 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:53:30.082151 (+ 3131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:53:30.082155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:53:30.082184 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:53:30.082696 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:53:30.082710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.084250 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.084258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:53:30.086425 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:53:30.086441 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.089079 (+ 2638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.089085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:53:30.089242 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:30.089267 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 04:53:30.089349 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 04:53:30.089461 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":643,"threads_started":1}
I20260806 04:53:30.090566 26007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:30.078328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.13.148.131:35879)
0806 04:53:30.081483 (+ 3155us) server_negotiation.cc:206] Beginning negotiation
0806 04:53:30.081492 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:53:30.081534 (+ 42us) server_negotiation.cc:407] Connection header received
0806 04:53:30.081626 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:53:30.081651 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:53:30.081885 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:53:30.082133 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:53:30.082989 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.084113 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.086861 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.088563 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.088751 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:30.088939 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 04:53:30.090138 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0806 04:53:30.090281 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2955,"thread_start_us":2699,"threads_started":1}
I20260806 04:53:30.092089 25790 tablet_service.cc:1467] Processing CreateTablet for tablet 2996e5c368a14c42ac414d98bad47d20 (DEFAULT_TABLE table=test-table [id=ce04729ce46d499092bd2d4947fff420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:53:30.092175 25644 tablet_service.cc:1467] Processing CreateTablet for tablet 2996e5c368a14c42ac414d98bad47d20 (DEFAULT_TABLE table=test-table [id=ce04729ce46d499092bd2d4947fff420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:53:30.093591 25790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2996e5c368a14c42ac414d98bad47d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:53:30.093626 25644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2996e5c368a14c42ac414d98bad47d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:53:30.094053 25926 tablet_service.cc:1467] Processing CreateTablet for tablet 2996e5c368a14c42ac414d98bad47d20 (DEFAULT_TABLE table=test-table [id=ce04729ce46d499092bd2d4947fff420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:53:30.095379 25926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2996e5c368a14c42ac414d98bad47d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:53:30.104681 26010 tablet_bootstrap.cc:492] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b: Bootstrap starting.
I20260806 04:53:30.108306 26010 tablet_bootstrap.cc:654] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b: Neither blocks nor log segments found. Creating new log.
I20260806 04:53:30.109217 26012 tablet_bootstrap.cc:492] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92: Bootstrap starting.
I20260806 04:53:30.109648 26010 log.cc:824] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b: Log is configured to *not* fsync() on all Append() calls
I20260806 04:53:30.111495 26011 tablet_bootstrap.cc:492] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77: Bootstrap starting.
I20260806 04:53:30.112557 26012 tablet_bootstrap.cc:654] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92: Neither blocks nor log segments found. Creating new log.
I20260806 04:53:30.112823 26010 tablet_bootstrap.cc:492] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b: No bootstrap required, opened a new log
I20260806 04:53:30.113098 26010 ts_tablet_manager.cc:1397] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260806 04:53:30.113626 26012 log.cc:824] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92: Log is configured to *not* fsync() on all Append() calls
I20260806 04:53:30.114805 26011 tablet_bootstrap.cc:654] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77: Neither blocks nor log segments found. Creating new log.
I20260806 04:53:30.116305 26012 tablet_bootstrap.cc:492] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92: No bootstrap required, opened a new log
I20260806 04:53:30.116464 26011 log.cc:824] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77: Log is configured to *not* fsync() on all Append() calls
I20260806 04:53:30.116627 26012 ts_tablet_manager.cc:1397] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 04:53:30.119673 26011 tablet_bootstrap.cc:492] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77: No bootstrap required, opened a new log
I20260806 04:53:30.119915 26011 ts_tablet_manager.cc:1397] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 04:53:30.123304 26010 raft_consensus.cc:348] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a18364c4a476e958e613925f4ba2b" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 39713 } } peers { permanent_uuid: "594d369b58964892a79392adede5fb92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35879 } } peers { permanent_uuid: "0d2bae01a25e4186a159900cfca97e77" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37319 } }
I20260806 04:53:30.123734 26010 raft_consensus.cc:374] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:53:30.123837 26010 raft_consensus.cc:729] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f04a18364c4a476e958e613925f4ba2b, State: Initialized, Role: FOLLOWER
I20260806 04:53:30.123731 26012 raft_consensus.cc:348] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a18364c4a476e958e613925f4ba2b" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 39713 } } peers { permanent_uuid: "594d369b58964892a79392adede5fb92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35879 } } peers { permanent_uuid: "0d2bae01a25e4186a159900cfca97e77" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37319 } }
I20260806 04:53:30.124238 26012 raft_consensus.cc:374] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:53:30.124341 26012 raft_consensus.cc:729] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594d369b58964892a79392adede5fb92, State: Initialized, Role: FOLLOWER
I20260806 04:53:30.124421 26010 consensus_queue.cc:260] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a18364c4a476e958e613925f4ba2b" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 39713 } } peers { permanent_uuid: "594d369b58964892a79392adede5fb92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35879 } } peers { permanent_uuid: "0d2bae01a25e4186a159900cfca97e77" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37319 } }
I20260806 04:53:30.124907 26012 consensus_queue.cc:260] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a18364c4a476e958e613925f4ba2b" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 39713 } } peers { permanent_uuid: "594d369b58964892a79392adede5fb92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35879 } } peers { permanent_uuid: "0d2bae01a25e4186a159900cfca97e77" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37319 } }
I20260806 04:53:30.127619 25991 heartbeater.cc:503] Master 127.13.148.190:42373 was elected leader, sending a full tablet report...
I20260806 04:53:30.127718 26010 ts_tablet_manager.cc:1428] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
I20260806 04:53:30.128170 25855 heartbeater.cc:503] Master 127.13.148.190:42373 was elected leader, sending a full tablet report...
I20260806 04:53:30.128269 26012 ts_tablet_manager.cc:1428] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260806 04:53:30.130311 26011 raft_consensus.cc:348] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a18364c4a476e958e613925f4ba2b" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 39713 } } peers { permanent_uuid: "594d369b58964892a79392adede5fb92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35879 } } peers { permanent_uuid: "0d2bae01a25e4186a159900cfca97e77" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37319 } }
I20260806 04:53:30.130755 26011 raft_consensus.cc:374] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:53:30.130856 26011 raft_consensus.cc:729] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2bae01a25e4186a159900cfca97e77, State: Initialized, Role: FOLLOWER
I20260806 04:53:30.131397 26011 consensus_queue.cc:260] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a18364c4a476e958e613925f4ba2b" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 39713 } } peers { permanent_uuid: "594d369b58964892a79392adede5fb92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35879 } } peers { permanent_uuid: "0d2bae01a25e4186a159900cfca97e77" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37319 } }
I20260806 04:53:30.132994 26011 ts_tablet_manager.cc:1428] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260806 04:53:30.182184 26018 raft_consensus.cc:482] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:53:30.182476 26018 raft_consensus.cc:504] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a18364c4a476e958e613925f4ba2b" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 39713 } } peers { permanent_uuid: "594d369b58964892a79392adede5fb92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35879 } } peers { permanent_uuid: "0d2bae01a25e4186a159900cfca97e77" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37319 } }
I20260806 04:53:30.184046 26018 leader_election.cc:288] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f04a18364c4a476e958e613925f4ba2b (127.13.148.130:39713), 594d369b58964892a79392adede5fb92 (127.13.148.131:35879)
I20260806 04:53:30.191213 26005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:30.184466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:44551 (local address 127.13.148.130:39713)
0806 04:53:30.184681 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 04:53:30.184686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:53:30.185238 (+ 552us) server_negotiation.cc:407] Connection header received
0806 04:53:30.185441 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:53:30.185446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:53:30.185533 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:53:30.185625 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:53:30.186484 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.187533 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.189546 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.190320 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.190423 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:30.190476 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 04:53:30.190857 (+ 381us) server_negotiation.cc:299] Negotiation successful
0806 04:53:30.190976 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 04:53:30.191298 26007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:30.185520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:33351 (local address 127.13.148.131:35879)
0806 04:53:30.185716 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 04:53:30.185721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:53:30.186005 (+ 284us) server_negotiation.cc:407] Connection header received
0806 04:53:30.186138 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:53:30.186143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:53:30.186217 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:53:30.186326 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:53:30.186991 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.187879 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.189697 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.190461 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.190597 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:30.190659 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 04:53:30.190960 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 04:53:30.191077 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 04:53:30.191305 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:30.184384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:39713 (local address 127.13.148.129:44551)
0806 04:53:30.185117 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0806 04:53:30.185144 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 04:53:30.185289 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:53:30.185800 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:53:30.185803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:53:30.185824 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:53:30.186326 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:53:30.186339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.187673 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.187679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:53:30.189399 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:53:30.189409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.190511 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.190519 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:53:30.190686 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:30.190726 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 04:53:30.190794 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 04:53:30.190873 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":57,"spinlock_wait_cycles":20224,"thread_start_us":415,"threads_started":1}
I20260806 04:53:30.191512 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:30.185254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:35879 (local address 127.13.148.129:33351)
0806 04:53:30.185867 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0806 04:53:30.185894 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 04:53:30.186020 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:53:30.186341 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:53:30.186343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:53:30.186359 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:53:30.186797 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:53:30.186810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.188031 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.188034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:53:30.189564 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:53:30.189577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.190667 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.190671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:53:30.190796 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:30.190829 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 04:53:30.190913 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 04:53:30.191021 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":363,"threads_started":1}
I20260806 04:53:30.192420 25810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2996e5c368a14c42ac414d98bad47d20" candidate_uuid: "0d2bae01a25e4186a159900cfca97e77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04a18364c4a476e958e613925f4ba2b" is_pre_election: true
I20260806 04:53:30.192456 25946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2996e5c368a14c42ac414d98bad47d20" candidate_uuid: "0d2bae01a25e4186a159900cfca97e77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "594d369b58964892a79392adede5fb92" is_pre_election: true
I20260806 04:53:30.192909 25810 raft_consensus.cc:2454] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d2bae01a25e4186a159900cfca97e77 in term 0.
I20260806 04:53:30.192927 25946 raft_consensus.cc:2454] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d2bae01a25e4186a159900cfca97e77 in term 0.
I20260806 04:53:30.193534 25596 leader_election.cc:302] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d2bae01a25e4186a159900cfca97e77, f04a18364c4a476e958e613925f4ba2b; no voters:
I20260806 04:53:30.193878 26018 raft_consensus.cc:2781] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:53:30.194020 26018 raft_consensus.cc:482] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:53:30.194125 26018 raft_consensus.cc:3037] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:53:30.196049 26018 raft_consensus.cc:504] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a18364c4a476e958e613925f4ba2b" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 39713 } } peers { permanent_uuid: "594d369b58964892a79392adede5fb92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35879 } } peers { permanent_uuid: "0d2bae01a25e4186a159900cfca97e77" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37319 } }
I20260806 04:53:30.196616 26018 leader_election.cc:288] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [CANDIDATE]: Term 1 election: Requested vote from peers f04a18364c4a476e958e613925f4ba2b (127.13.148.130:39713), 594d369b58964892a79392adede5fb92 (127.13.148.131:35879)
I20260806 04:53:30.196991 25810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2996e5c368a14c42ac414d98bad47d20" candidate_uuid: "0d2bae01a25e4186a159900cfca97e77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04a18364c4a476e958e613925f4ba2b"
I20260806 04:53:30.197146 25946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2996e5c368a14c42ac414d98bad47d20" candidate_uuid: "0d2bae01a25e4186a159900cfca97e77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "594d369b58964892a79392adede5fb92"
I20260806 04:53:30.197206 25810 raft_consensus.cc:3037] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:53:30.197365 25946 raft_consensus.cc:3037] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:53:30.199460 25946 raft_consensus.cc:2454] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d2bae01a25e4186a159900cfca97e77 in term 1.
I20260806 04:53:30.199481 25810 raft_consensus.cc:2454] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d2bae01a25e4186a159900cfca97e77 in term 1.
I20260806 04:53:30.199846 25596 leader_election.cc:302] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d2bae01a25e4186a159900cfca97e77, f04a18364c4a476e958e613925f4ba2b; no voters:
I20260806 04:53:30.200079 26018 raft_consensus.cc:2781] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:53:30.200459 26018 raft_consensus.cc:686] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [term 1 LEADER]: Becoming Leader. State: Replica: 0d2bae01a25e4186a159900cfca97e77, State: Running, Role: LEADER
I20260806 04:53:30.200932 26018 consensus_queue.cc:237] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a18364c4a476e958e613925f4ba2b" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 39713 } } peers { permanent_uuid: "594d369b58964892a79392adede5fb92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35879 } } peers { permanent_uuid: "0d2bae01a25e4186a159900cfca97e77" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37319 } }
W20260806 04:53:30.202059 25712 tablet.cc:2369] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:53:30.205247 25489 catalog_manager.cc:5495] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d2bae01a25e4186a159900cfca97e77 (127.13.148.129). New cstate: current_term: 1 leader_uuid: "0d2bae01a25e4186a159900cfca97e77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a18364c4a476e958e613925f4ba2b" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 39713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "594d369b58964892a79392adede5fb92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d2bae01a25e4186a159900cfca97e77" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 37319 } health_report { overall_health: HEALTHY } } }
W20260806 04:53:30.215962 25856 tablet.cc:2369] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:53:30.238302 26009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:30.233577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.13.148.129:37319)
0806 04:53:30.233740 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 04:53:30.233745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:53:30.233786 (+ 41us) server_negotiation.cc:407] Connection header received
0806 04:53:30.233961 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:53:30.233964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:53:30.234038 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:53:30.234147 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:53:30.234482 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.235161 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.236429 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.236905 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.236999 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:30.237429 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:53:30.237920 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:53:30.238019 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 04:53:30.238081 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 04:53:30.238136 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:53:30.246435 25810 raft_consensus.cc:1261] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b [term 1 FOLLOWER]: Refusing update from remote peer 0d2bae01a25e4186a159900cfca97e77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:53:30.246435 25946 raft_consensus.cc:1261] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92 [term 1 FOLLOWER]: Refusing update from remote peer 0d2bae01a25e4186a159900cfca97e77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:53:30.247370 26021 consensus_queue.cc:1035] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f04a18364c4a476e958e613925f4ba2b" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 39713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:53:30.247748 26018 consensus_queue.cc:1035] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "594d369b58964892a79392adede5fb92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 35879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 04:53:30.258373 25992 tablet.cc:2369] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:53:30.262991 26026 mvcc.cc:205] Tried to move back new op lower bound from 7315423273959284736 to 7315423273786179584. Current Snapshot: MvccSnapshot[applied={T|T < 7315423273959284736}]
I20260806 04:53:30.275094 25489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:53:30.275350 25489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:53:30.278738 25489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7c41fb34c4f48e185876476856aea50: Sending DeleteTablet for 3 replicas of tablet 2996e5c368a14c42ac414d98bad47d20
I20260806 04:53:30.279686 25790 tablet_service.cc:1514] Processing DeleteTablet for tablet 2996e5c368a14c42ac414d98bad47d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:53:30 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:52964
I20260806 04:53:30.279866 25926 tablet_service.cc:1514] Processing DeleteTablet for tablet 2996e5c368a14c42ac414d98bad47d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:53:30 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:44694
I20260806 04:53:30.280552 25644 tablet_service.cc:1514] Processing DeleteTablet for tablet 2996e5c368a14c42ac414d98bad47d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:53:30 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:47620
I20260806 04:53:30.282522 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 25575
I20260806 04:53:30.295706 26035 tablet_replica.cc:330] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b: stopping tablet replica
I20260806 04:53:30.296141 26035 raft_consensus.cc:2229] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:53:30.296695 26035 raft_consensus.cc:2258] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:53:30.300531 26035 ts_tablet_manager.cc:1905] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:53:30.303864 26036 tablet_replica.cc:330] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92: stopping tablet replica
I20260806 04:53:30.304296 26036 raft_consensus.cc:2229] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:53:30.304917 26036 raft_consensus.cc:2258] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:53:30.307053 26036 ts_tablet_manager.cc:1905] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:53:30.307477 26037 tablet_replica.cc:330] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77: stopping tablet replica
I20260806 04:53:30.307569 26035 ts_tablet_manager.cc:1918] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:53:30.307729 26035 log.cc:1196] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/2996e5c368a14c42ac414d98bad47d20
I20260806 04:53:30.307891 26037 raft_consensus.cc:2229] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:53:30.308318 26035 ts_tablet_manager.cc:1939] T 2996e5c368a14c42ac414d98bad47d20 P f04a18364c4a476e958e613925f4ba2b: Deleting consensus metadata
I20260806 04:53:30.308689 26037 raft_consensus.cc:2258] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:53:30.310288 25474 catalog_manager.cc:4841] TS f04a18364c4a476e958e613925f4ba2b (127.13.148.130:39713): tablet 2996e5c368a14c42ac414d98bad47d20 (table test-table [id=ce04729ce46d499092bd2d4947fff420]) successfully deleted
I20260806 04:53:30.312273 26037 ts_tablet_manager.cc:1905] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:53:30.313815 26036 ts_tablet_manager.cc:1918] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:53:30.313962 26036 log.cc:1196] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/2996e5c368a14c42ac414d98bad47d20
I20260806 04:53:30.314462 26036 ts_tablet_manager.cc:1939] T 2996e5c368a14c42ac414d98bad47d20 P 594d369b58964892a79392adede5fb92: Deleting consensus metadata
Aug 06 04:53:30 dist-test-slave-1x4g krb5kdc[25004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:53:30.316179 25474 catalog_manager.cc:4841] TS 594d369b58964892a79392adede5fb92 (127.13.148.131:35879): tablet 2996e5c368a14c42ac414d98bad47d20 (table test-table [id=ce04729ce46d499092bd2d4947fff420]) successfully deleted
I20260806 04:53:30.320498 26037 ts_tablet_manager.cc:1918] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:53:30.320673 26037 log.cc:1196] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/2996e5c368a14c42ac414d98bad47d20
I20260806 04:53:30.321259 26037 ts_tablet_manager.cc:1939] T 2996e5c368a14c42ac414d98bad47d20 P 0d2bae01a25e4186a159900cfca97e77: Deleting consensus metadata
I20260806 04:53:30.322780 25474 catalog_manager.cc:4841] TS 0d2bae01a25e4186a159900cfca97e77 (127.13.148.129:37319): tablet 2996e5c368a14c42ac414d98bad47d20 (table test-table [id=ce04729ce46d499092bd2d4947fff420]) successfully deleted
I20260806 04:53:30.324765 26009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:30.284091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47628 (local address 127.13.148.129:37319)
0806 04:53:30.284277 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 04:53:30.284284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:53:30.303260 (+ 18976us) server_negotiation.cc:407] Connection header received
0806 04:53:30.303435 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:53:30.303439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:53:30.303515 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:53:30.303606 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:53:30.305352 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.307084 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.310821 (+ 3737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.311487 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.311584 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:30.317282 (+ 5698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:53:30.317303 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:53:30.317323 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:53:30.317370 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:53:30.321744 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:53:30.322621 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:53:30.322630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:53:30.322633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:53:30.322800 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:53:30.323224 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:53:30.323229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:53:30.323231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:53:30.323689 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:53:30.323967 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 04:53:30.324196 (+ 229us) server_negotiation.cc:299] Negotiation successful
0806 04:53:30.324427 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:53:30.325564 25705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:53:30.419292 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 25575
I20260806 04:53:30.439116 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 25717
Aug 06 04:53:30 dist-test-slave-1x4g krb5kdc[25004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:53:30.453150 26005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:30.439792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.13.148.130:39713)
0806 04:53:30.439972 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 04:53:30.439976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:53:30.440006 (+ 30us) server_negotiation.cc:407] Connection header received
0806 04:53:30.440166 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:53:30.440169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:53:30.440226 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:53:30.440319 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:53:30.441608 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.442400 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.443648 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.444018 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.444114 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:30.448159 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:53:30.448170 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:53:30.448183 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:53:30.448210 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:53:30.451038 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:53:30.451566 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:53:30.451573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:53:30.451575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:53:30.451684 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:53:30.451983 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:53:30.451986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:53:30.451988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:53:30.452296 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:53:30.452485 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 04:53:30.452665 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 04:53:30.452918 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 04:53:30.453766 25850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:53:30.537348 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 25717
I20260806 04:53:30.555483 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 25861
Aug 06 04:53:30 dist-test-slave-1x4g krb5kdc[25004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
I20260806 04:53:30.569106 26007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:53:30.556257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44702 (local address 127.13.148.131:35879)
0806 04:53:30.556412 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 04:53:30.556416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:53:30.556432 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:53:30.556630 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:53:30.556634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:53:30.556697 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:53:30.556857 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:53:30.558258 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.558878 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.560062 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:53:30.560420 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:53:30.560500 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:53:30.564262 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:53:30.564274 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:53:30.564289 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:53:30.564314 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:53:30.566800 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:53:30.567385 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:53:30.567392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:53:30.567394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:53:30.567508 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:53:30.567821 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:53:30.567823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:53:30.567825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:53:30.568160 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:53:30.568360 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 04:53:30.568569 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 04:53:30.568710 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:53:30.569701 25986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:53:30.651067 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 25861
I20260806 04:53:30.668396 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 25458
I20260806 04:53:30.669632 25521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:53:30.773197 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 25458
I20260806 04:53:30.797688 13906 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 04:53:31.133796 13906 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 04:53:31.134089 13906 mini_ranger.cc:66] Stopping Ranger...
I20260806 04:53:31.479414 13906 mini_ranger.cc:68] Stopped Ranger
2026-08-06 04:53:31.479 UTC [25034] LOG: received smart shutdown request
2026-08-06 04:53:31.483 UTC [25034] LOG: background worker "logical replication launcher" (PID 25044) exited with exit code 1
2026-08-06 04:53:31.483 UTC [25039] LOG: shutting down
2026-08-06 04:53:31.610 UTC [25034] LOG: database system is shut down
2026-08-06T04:53:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44805 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:53:31 dist-test-slave-1x4g krb5kdc[26051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:53:31 dist-test-slave-1x4g krb5kdc[26051](info): set up 2 sockets
Aug 06 04:53:31 dist-test-slave-1x4g krb5kdc[26051](info): commencing operation
krb5kdc: starting...
W20260806 04:53:32.761615 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:53:32 dist-test-slave-1x4g krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:53:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:53:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.148.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.148.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.148.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/rangeradmin_127.13.148.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.148.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/rangeradmin_127.13.148.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.148.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.148.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.148.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/rangerlookup_127.13.148.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.148.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/rangerlookup_127.13.148.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.148.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.148.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.148.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/HTTP_127.13.148.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.148.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/HTTP_127.13.148.148@KRBTEST.COM.keytab.
I20260806 04:53:33.077725 13906 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task10hk37/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/postgres -l logfile start
2026-08-06 04:53:35.292 UTC [26081] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 04:53:35.292 UTC [26081] LOG: listening on IPv4 address "127.13.148.148", port 34319
2026-08-06 04:53:35.295 UTC [26081] LOG: listening on Unix socket "/tmp/.s.PGSQL.34319"
2026-08-06 04:53:35.314 UTC [26084] LOG: database system was shut down at 2026-08-06 04:53:34 UTC
2026-08-06 04:53:35.317 UTC [26081] LOG: database system is ready to accept connections
I20260806 04:53:36.250634 13906 mini_postgres.cc:96] Postgres bound to 34319
2026-08-06 04:53:36.270 UTC [26092] FATAL: database "slave" does not exist
127.13.148.148:34319 - accepting connections
I20260806 04:53:36.270838 13906 mini_ranger.cc:161] Starting Ranger...
I20260806 04:53:36.296599 13906 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 04:53:36.698580 13906 mini_ranger.cc:87] Created ranger Postgres database
I20260806 04:53:36.698683 13906 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-admin
2026-08-06 04:53:36,966 [I] DB FLAVOR :POSTGRES
2026-08-06 04:53:36,967 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 04:53:36,967 [I] Checking connection..
2026-08-06 04:53:36,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 04:53:37,212 [I] Checking connection passed.
2026-08-06 04:53:37,212 [I] --------- Verifying version history table ---------
2026-08-06 04:53:37,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:53:37,420 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 04:53:37,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:53:37,654 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 04:53:37,654 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 04:53:37,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 04:53:37.878 UTC [26158] WARNING: there is no transaction in progress
2026-08-06 04:53:37,884 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 04:53:37,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:53:38,091 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 04:53:38,092 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 04:53:38,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 04:53:38,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 04:53:38,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1x4g.c.gcp-upstream.internal','N') ;"
2026-08-06 04:53:38,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 04:53:38,954 [I] Table x_portal_user does not exist in database ranger
2026-08-06 04:53:38,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 04:53:39,190 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 04:53:39,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 04:53:39,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 04:53:39,660 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 04:53:39,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 04:53:40.497 UTC [26269] WARNING: there is no transaction in progress
2026-08-06 04:53:40.512 UTC [26269] WARNING: there is no transaction in progress
2026-08-06 04:53:40.525 UTC [26269] WARNING: there is no transaction in progress
2026-08-06 04:53:40.537 UTC [26269] WARNING: there is no transaction in progress
2026-08-06 04:53:40.553 UTC [26269] WARNING: there is no transaction in progress
2026-08-06 04:53:40.570 UTC [26269] WARNING: there is no transaction in progress
2026-08-06 04:53:40.620 UTC [26269] WARNING: there is no transaction in progress
2026-08-06 04:53:40.625 UTC [26269] WARNING: there is no transaction in progress
2026-08-06 04:53:40.637 UTC [26269] WARNING: there is no transaction in progress
2026-08-06 04:53:40.645 UTC [26269] WARNING: there is no transaction in progress
2026-08-06 04:53:40.654 UTC [26269] WARNING: there is no transaction in progress
2026-08-06 04:53:40.665 UTC [26269] WARNING: there is no transaction in progress
2026-08-06 04:53:40.673 UTC [26269] WARNING: there is no transaction in progress
2026-08-06 04:53:40.678 UTC [26269] WARNING: there is no transaction in progress
2026-08-06 04:53:40.683 UTC [26269] WARNING: there is no transaction in progress
2026-08-06 04:53:40.689 UTC [26269] WARNING: there is no transaction in progress
2026-08-06 04:53:41,204 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 04:53:41,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 04:53:41,423 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 04:53:41,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 04:53:41,660 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 04:53:41,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 04:53:41,885 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 04:53:41,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 04:53:42,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 04:53:42,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1x4g.c.gcp-upstream.internal';"
2026-08-06 04:53:42,541 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 04:53:42,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 04:53:42,768 [I] DB_PATCHES have already been applied
I20260806 04:53:42.774153 13906 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-admin:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task10hk37/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 04:53:42.774245 13906 mini_ranger.cc:193] Using host: 127.13.148.148
I20260806 04:53:42.790145 13906 mini_ranger.cc:227] Ranger admin URL: http://127.13.148.148:50993
Aug 06, 2026 4:53:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:53:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:53:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 4:53:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 4:53:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 4:53:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 4:53:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 04:53:43.860241 13906 mini_ranger.cc:160] Time spent starting Ranger: real 7.589s user 0.000s sys 0.043s
Aug 06, 2026 4:54:02 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 4:54:02 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Aug 06, 2026 4:54:02 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 4:54:02 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 4:54:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 4:54:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 4:54:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 4:54:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 4:54:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 4:54:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 4:54:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 4:54:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 4:54:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 4:54:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 4:54:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 4:54:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 4:54:02 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 4:54:03 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 04:54:04.176065 13906 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.148.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.148.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.148.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/rangerkms_127.13.148.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.148.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/rangerkms_127.13.148.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.148.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.148.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.148.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/HTTP_127.13.148.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.148.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/HTTP_127.13.148.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 04:54:04 dist-test-slave-1x4g krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992044, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 04:54:04.327721 13906 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 04:54:04.346508 13906 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 04:54:05.106618 13906 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 04:54:05.106737 13906 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-kms
I20260806 04:54:05.106781 13906 mini_ranger_kms.cc:227] Using postgres at 127.13.148.148:34319
2026-08-06 04:54:05,449 [I] DB FLAVOR :POSTGRES
2026-08-06 04:54:05,450 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 04:54:05,450 [I] Checking connection
2026-08-06 04:54:05,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 04:54:05,672 [I] connection success
2026-08-06 04:54:05,672 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 04:54:05,672 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 04:54:05,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 04:54:05,895 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 04:54:05,895 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 04:54:05,896 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 04:54:05,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.148:34319/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 04:54:06,137 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 04:54:06.569692 13906 mini_ranger_kms.cc:310] Created kms service
I20260806 04:54:06.765712 13906 mini_ranger_kms.cc:326] Created kudu user
I20260806 04:54:06.806979 13906 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 04:54:06 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992044, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
I20260806 04:54:07.164438 13906 mini_ranger_kms.cc:384] Added ranger policy
I20260806 04:54:07.164767 13906 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-kms:/tmp/dist-test-task10hk37/build/asan/bin/postgresql.jar:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task10hk37/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 04:54:07.164839 13906 mini_ranger_kms.cc:240] Using host: 127.13.148.148
I20260806 04:54:07.187006 13906 mini_ranger_kms.cc:276] Ranger KMS PID: 26452
I20260806 04:54:07.187146 13906 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.148.148:59515
Aug 06, 2026 4:54:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:54:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:54:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 4:54:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 4:54:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 4:54:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 04:54:09.316180 13906 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.988s user 0.087s sys 0.275s
I20260806 04:54:09.316414 13906 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 04:54:09.316581 13906 mini_ranger_kms.cc:401] 127.13.148.148:59515/kms/v1/keys
Aug 06 04:54:09 dist-test-slave-1x4g krb5kdc[26051](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785992049, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 04:54:09 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992049, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.148@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
Aug 06 04:54:10 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992049, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.148@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
Aug 06 04:54:12 dist-test-slave-1x4g krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:54:12.469334 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:37709
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:37709
--ranger_config_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.148:59515/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:54:12.772759 26504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:12.773132 26504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:12.773295 26504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:12.779907 26504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:54:12.780052 26504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:12.780154 26504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:12.780256 26504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:54:12.780344 26504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:54:12.789286 26504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.148.148:59515/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:37709
--ranger_config_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:37709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:12.790486 26504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:12.791726 26504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:12.804381 26512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:12.804373 26510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:12.804813 26509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:12.805980 26504 server_base.cc:1034] running on GCE node
I20260806 04:54:12.806928 26504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:12.808523 26504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:12.809796 26504 hybrid_clock.cc:648] HybridClock initialized: now 1785992052809751 us; error 42 us; skew 500 ppm
I20260806 04:54:12.810226 26504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:12 dist-test-slave-1x4g krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992052, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:12.818699 26504 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:54:12.820194 26504 webserver.cc:460] Webserver started at http://127.13.148.190:43683/ using document root <none> and password file <none>
I20260806 04:54:12.820901 26504 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:12.821023 26504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:12.821307 26504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:54:12 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992052, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
I20260806 04:54:12.882629 26504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "f183e8427aa34d1a812331a828cdb147"
format_stamp: "Formatted at 2026-08-06 04:54:12 on dist-test-slave-1x4g"
server_key: "0ffba0f54cbca6da0e82084277b96c05"
server_key_iv: "95c9fc4765f45e9814c5ef0997344962"
server_key_version: "kuduclusterkey@0"
I20260806 04:54:12.883491 26504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "f183e8427aa34d1a812331a828cdb147"
format_stamp: "Formatted at 2026-08-06 04:54:12 on dist-test-slave-1x4g"
server_key: "0ffba0f54cbca6da0e82084277b96c05"
server_key_iv: "95c9fc4765f45e9814c5ef0997344962"
server_key_version: "kuduclusterkey@0"
I20260806 04:54:12.889009 26504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 04:54:12.908049 26519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:12.909461 26504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:54:12.909651 26504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "f183e8427aa34d1a812331a828cdb147"
format_stamp: "Formatted at 2026-08-06 04:54:12 on dist-test-slave-1x4g"
server_key: "0ffba0f54cbca6da0e82084277b96c05"
server_key_iv: "95c9fc4765f45e9814c5ef0997344962"
server_key_version: "kuduclusterkey@0"
I20260806 04:54:12.909874 26504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:12.931146 26504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:12.934866 26504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:12.935192 26504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:12.953615 26504 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:37709
I20260806 04:54:12.953619 26571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:37709 every 8 connection(s)
I20260806 04:54:12.954795 26504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:54:12.958442 26572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:12.959419 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 26504
I20260806 04:54:12.959657 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:54:12.960585 13906 external_mini_cluster.cc:1428] Setting key 25d18adf66968cf024a822685d93462f
I20260806 04:54:12.974359 26572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147: Bootstrap starting.
I20260806 04:54:12.978653 26572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:12.979772 26572 log.cc:824] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:12.982877 26572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147: No bootstrap required, opened a new log
I20260806 04:54:12.991277 26572 raft_consensus.cc:348] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183e8427aa34d1a812331a828cdb147" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37709 } }
I20260806 04:54:12.991714 26572 raft_consensus.cc:374] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:12.991839 26572 raft_consensus.cc:729] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f183e8427aa34d1a812331a828cdb147, State: Initialized, Role: FOLLOWER
Aug 06 04:54:12 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:12.992457 26572 consensus_queue.cc:260] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183e8427aa34d1a812331a828cdb147" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37709 } }
I20260806 04:54:12.992708 26572 raft_consensus.cc:388] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:54:12.992868 26572 raft_consensus.cc:482] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:54:12.993028 26572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:12.995879 26572 raft_consensus.cc:504] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183e8427aa34d1a812331a828cdb147" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37709 } }
I20260806 04:54:12.996400 26572 leader_election.cc:302] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f183e8427aa34d1a812331a828cdb147; no voters:
I20260806 04:54:12.997841 26572 leader_election.cc:288] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:54:12.997980 26577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:54:13.004143 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:12.962812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.13.148.190:37709)
0806 04:54:12.978157 (+ 15345us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:12.978174 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:12.978229 (+ 55us) server_negotiation.cc:407] Connection header received
0806 04:54:12.980949 (+ 2720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:12.980978 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:12.981469 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:12.981957 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:12.983471 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:12.985606 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:12.987300 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:12.988486 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:12.988686 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:12.993672 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:12.993681 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:12.993701 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:12.993748 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:12.997585 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:12.998508 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:12.998519 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:12.998530 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:12.998729 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:12.999240 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:12.999247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:12.999252 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:13.000126 (+ 874us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:13.001836 (+ 1710us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:13.002146 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 04:54:13.002599 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15074,"thread_start_us":15657,"threads_started":1}
I20260806 04:54:13.006166 26572 sys_catalog.cc:564] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:54:13.008926 26577 raft_consensus.cc:686] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [term 1 LEADER]: Becoming Leader. State: Replica: f183e8427aa34d1a812331a828cdb147, State: Running, Role: LEADER
I20260806 04:54:13.009605 26577 consensus_queue.cc:237] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183e8427aa34d1a812331a828cdb147" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37709 } }
I20260806 04:54:13.012575 26578 sys_catalog.cc:455] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f183e8427aa34d1a812331a828cdb147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183e8427aa34d1a812331a828cdb147" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37709 } } }
I20260806 04:54:13.012980 26578 sys_catalog.cc:458] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [sys.catalog]: This master's current role is: LEADER
I20260806 04:54:13.013674 26572 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 04:54:13.014295 26579 sys_catalog.cc:455] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f183e8427aa34d1a812331a828cdb147. Latest consensus state: current_term: 1 leader_uuid: "f183e8427aa34d1a812331a828cdb147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183e8427aa34d1a812331a828cdb147" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37709 } } }
I20260806 04:54:13.014565 26579 sys_catalog.cc:458] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [sys.catalog]: This master's current role is: LEADER
I20260806 04:54:13.025882 26591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:54:13.038604 26591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:54:13.072820 26591 catalog_manager.cc:1347] Generated new cluster ID: 2427dc1163e04d0993c668bf4279c225
I20260806 04:54:13.072973 26591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:54:13.087900 26591 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:54:13.089921 26591 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:54:13.100732 26591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147: Generated new TSK 0
I20260806 04:54:13.101970 26591 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:54:13.841267 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.148:59515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:37709
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:54:14 dist-test-slave-1x4g krb5kdc[26051](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785992054, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 04:54:14.248530 26617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:14.248906 26617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:14.249051 26617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:14.259374 26617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:14.259492 26617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:14.259713 26617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:54:14.273614 26617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.148.148:59515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:37709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:14.275236 26617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:14.276862 26617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:14.296057 26628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:14.296083 26629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:14.304987 26617 server_base.cc:1034] running on GCE node
W20260806 04:54:14.305676 26631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:14.306345 26617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:14.307322 26617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:14.308537 26617 hybrid_clock.cc:648] HybridClock initialized: now 1785992054308491 us; error 40 us; skew 500 ppm
I20260806 04:54:14.308923 26617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:14 dist-test-slave-1x4g krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992054, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:14.312256 26617 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:54:14.313989 26617 webserver.cc:460] Webserver started at http://127.13.148.129:42603/ using document root <none> and password file <none>
I20260806 04:54:14.314632 26617 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:14.314759 26617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:14.315052 26617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:54:14 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992054, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
I20260806 04:54:14.334385 26617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "a6d88d96cfa84d2b9482bac72f5cbdcc"
format_stamp: "Formatted at 2026-08-06 04:54:14 on dist-test-slave-1x4g"
server_key: "c0f024473464cefda23ad86c7604e6d3"
server_key_iv: "a08f3841d5c0681b8a3af859150b9708"
server_key_version: "kuduclusterkey@0"
I20260806 04:54:14.335522 26617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "a6d88d96cfa84d2b9482bac72f5cbdcc"
format_stamp: "Formatted at 2026-08-06 04:54:14 on dist-test-slave-1x4g"
server_key: "c0f024473464cefda23ad86c7604e6d3"
server_key_iv: "a08f3841d5c0681b8a3af859150b9708"
server_key_version: "kuduclusterkey@0"
I20260806 04:54:14.358456 26617 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.007s sys 0.010s
I20260806 04:54:14.393399 26638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:14.396202 26617 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.005s sys 0.015s
I20260806 04:54:14.396432 26617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "a6d88d96cfa84d2b9482bac72f5cbdcc"
format_stamp: "Formatted at 2026-08-06 04:54:14 on dist-test-slave-1x4g"
server_key: "c0f024473464cefda23ad86c7604e6d3"
server_key_iv: "a08f3841d5c0681b8a3af859150b9708"
server_key_version: "kuduclusterkey@0"
I20260806 04:54:14.396806 26617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:14.459592 26617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:14.464411 26617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:14.464823 26617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:14.472035 26617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:14.479965 26617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:54:14.480106 26617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:14.480293 26617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:54:14.480568 26617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:54:14 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992054, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
I20260806 04:54:14.710165 26617 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:41749
I20260806 04:54:14.710183 26751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:41749 every 8 connection(s)
I20260806 04:54:14.711601 26617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:54:14.718631 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 26617
I20260806 04:54:14.718840 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:54:14.719620 13906 external_mini_cluster.cc:1428] Setting key eada0e6d1e4ee4d78810f2465c2eccf9
Aug 06 04:54:14 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992054, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:14.741518 26754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:14.714573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:56151 (local address 127.13.148.190:37709)
0806 04:54:14.715116 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:14.715123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:14.716158 (+ 1035us) server_negotiation.cc:407] Connection header received
0806 04:54:14.719210 (+ 3052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:14.719217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:14.719318 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:14.719445 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:14.721723 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:14.722780 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:14.728326 (+ 5546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:14.729059 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:14.729186 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:14.733529 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:14.733535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:14.733540 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:14.733576 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:14.737109 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:14.738055 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:14.738060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:14.738063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:14.738198 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:14.738681 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:14.738686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:14.738688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:14.739031 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:14.739159 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:14.740834 (+ 1675us) server_negotiation.cc:299] Negotiation successful
0806 04:54:14.741083 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":283,"threads_started":1}
I20260806 04:54:14.743824 26755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:14.715136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:37709 (local address 127.13.148.129:56151)
0806 04:54:14.715955 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:14.716022 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:14.718876 (+ 2854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:14.719767 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:14.719782 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:14.720551 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:14.721547 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:14.721578 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:14.722968 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:14.722973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:14.724916 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:14.724933 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:14.729511 (+ 4578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:14.729521 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:14.729856 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:14.730716 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:14.730750 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:14.733295 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:14.737307 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:14.737316 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:14.737331 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:14.737923 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:14.738325 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:14.738329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:14.738332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:14.738550 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:14.739247 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:14.739253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:14.740340 (+ 1087us) client_negotiation.cc:769] Sending connection context
0806 04:54:14.742054 (+ 1714us) client_negotiation.cc:240] Negotiation successful
0806 04:54:14.742562 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":276,"threads_started":1}
I20260806 04:54:14.745949 26752 heartbeater.cc:347] Connected to a master server at 127.13.148.190:37709
I20260806 04:54:14.746260 26752 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:14.747079 26752 heartbeater.cc:511] Master 127.13.148.190:37709 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
I20260806 04:54:14.749735 26536 ts_manager.cc:194] Registered new tserver with Master: a6d88d96cfa84d2b9482bac72f5cbdcc (127.13.148.129:41749)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
I20260806 04:54:14.752100 26536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:56151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:54:14.783756 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.148:59515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:37709
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:54:14 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992054, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
W20260806 04:54:15.208443 26758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:15.208777 26758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:15.208976 26758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:15.216241 26758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:15.216446 26758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:15.216626 26758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:54:15.226130 26758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.148.148:59515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:37709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:15.227303 26758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:15.228469 26758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:15.242300 26773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:15.242652 26775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:15.242275 26772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:15.243573 26758 server_base.cc:1034] running on GCE node
I20260806 04:54:15.244065 26758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:15.244939 26758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:15.246137 26758 hybrid_clock.cc:648] HybridClock initialized: now 1785992055246071 us; error 63 us; skew 500 ppm
I20260806 04:54:15.246409 26758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:15 dist-test-slave-1x4g krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992055, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:15.249472 26758 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:54:15.251063 26758 webserver.cc:460] Webserver started at http://127.13.148.130:35981/ using document root <none> and password file <none>
I20260806 04:54:15.251818 26758 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:15.251942 26758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:15.252243 26758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:54:15 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992055, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
I20260806 04:54:15.268177 26758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "44cff7e36fef48c2b60bbde0dab7f926"
format_stamp: "Formatted at 2026-08-06 04:54:15 on dist-test-slave-1x4g"
server_key: "196f5dff746ffecd9b99927465192c53"
server_key_iv: "a289e61d19f43195512e28d31ee20093"
server_key_version: "kuduclusterkey@0"
I20260806 04:54:15.269205 26758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "44cff7e36fef48c2b60bbde0dab7f926"
format_stamp: "Formatted at 2026-08-06 04:54:15 on dist-test-slave-1x4g"
server_key: "196f5dff746ffecd9b99927465192c53"
server_key_iv: "a289e61d19f43195512e28d31ee20093"
server_key_version: "kuduclusterkey@0"
I20260806 04:54:15.275832 26758 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 04:54:15.292490 26782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:15.294066 26758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:54:15.294255 26758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "44cff7e36fef48c2b60bbde0dab7f926"
format_stamp: "Formatted at 2026-08-06 04:54:15 on dist-test-slave-1x4g"
server_key: "196f5dff746ffecd9b99927465192c53"
server_key_iv: "a289e61d19f43195512e28d31ee20093"
server_key_version: "kuduclusterkey@0"
I20260806 04:54:15.294529 26758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:15.325593 26758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:15.330298 26758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:15.330629 26758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:15.331696 26758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:15.333192 26758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:54:15.333276 26758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:15.333436 26758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:54:15.333491 26758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:15.375207 26758 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:41207
I20260806 04:54:15.375247 26895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:41207 every 8 connection(s)
I20260806 04:54:15.376751 26758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:54:15.377555 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 26758
I20260806 04:54:15.377719 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:54:15.378268 13906 external_mini_cluster.cc:1428] Setting key 334577d55e45d4e7b1b3b85e4f330679
Aug 06 04:54:15 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992055, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
I20260806 04:54:15.418606 26897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:15.381066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:34595 (local address 127.13.148.190:37709)
0806 04:54:15.401238 (+ 20172us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:15.401242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:15.401258 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:54:15.401306 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:15.401309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:15.401372 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:15.401510 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:15.403342 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:15.404463 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:15.406640 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:15.407139 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:15.407241 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:15.411624 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:15.411629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:15.411634 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:15.411668 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:15.415009 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:15.415817 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:15.415822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:15.415825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:15.415960 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:15.416419 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:15.416424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:15.416427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:15.416822 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:15.417002 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:15.417932 (+ 930us) server_negotiation.cc:299] Negotiation successful
0806 04:54:15.418172 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20063,"thread_start_us":19974,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
I20260806 04:54:15.419657 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:15.381756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:37709 (local address 127.13.148.130:34595)
0806 04:54:15.397236 (+ 15480us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:15.397287 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:15.399781 (+ 2494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:15.401752 (+ 1971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:15.401765 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:15.402351 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:15.403127 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:15.403152 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:15.404633 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:15.404637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:15.406476 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:15.406490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:15.407497 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:15.407503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:15.407786 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:15.408591 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:15.408616 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:15.411389 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:15.415157 (+ 3768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:15.415165 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:15.415181 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:15.415690 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:15.416071 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:15.416075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:15.416079 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:15.416299 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:15.417011 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:15.417019 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:15.417467 (+ 448us) client_negotiation.cc:769] Sending connection context
0806 04:54:15.417763 (+ 296us) client_negotiation.cc:240] Negotiation successful
0806 04:54:15.418297 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15168,"thread_start_us":14874,"threads_started":1}
I20260806 04:54:15.424923 26896 heartbeater.cc:347] Connected to a master server at 127.13.148.190:37709
I20260806 04:54:15.425298 26896 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:15.426406 26896 heartbeater.cc:511] Master 127.13.148.190:37709 requested a full tablet report, sending...
I20260806 04:54:15.428069 26536 ts_manager.cc:194] Registered new tserver with Master: 44cff7e36fef48c2b60bbde0dab7f926 (127.13.148.130:41207)
I20260806 04:54:15.429234 26536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:34595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
I20260806 04:54:15.456005 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.148:59515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:37709
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:54:15.755506 26752 heartbeater.cc:503] Master 127.13.148.190:37709 was elected leader, sending a full tablet report...
W20260806 04:54:15.807754 26902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:15.807986 26902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:15.808260 26902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:15.815058 26902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:15.815194 26902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:15.815447 26902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:54:15.824936 26902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.148.148:59515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:37709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:15.826253 26902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:15.827651 26902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:15.840833 26907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:15.840888 26908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:15.841019 26910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:15.841693 26902 server_base.cc:1034] running on GCE node
I20260806 04:54:15.842212 26902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:15.843057 26902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:15.844241 26902 hybrid_clock.cc:648] HybridClock initialized: now 1785992055844205 us; error 28 us; skew 500 ppm
I20260806 04:54:15.844551 26902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:15 dist-test-slave-1x4g krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992055, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:15.847564 26902 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:54:15.849088 26902 webserver.cc:460] Webserver started at http://127.13.148.131:41835/ using document root <none> and password file <none>
I20260806 04:54:15.849710 26902 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:15.849865 26902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:15.850169 26902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:54:15 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992055, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
I20260806 04:54:15.867606 26902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "4fbd57011b1a4b08938c4daef6bf1068"
format_stamp: "Formatted at 2026-08-06 04:54:15 on dist-test-slave-1x4g"
server_key: "5bfb0ea02f69c61edbb5c9494689aefe"
server_key_iv: "cdbb02d07f4db337c4b5b5d545ee8831"
server_key_version: "kuduclusterkey@0"
I20260806 04:54:15.868626 26902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "4fbd57011b1a4b08938c4daef6bf1068"
format_stamp: "Formatted at 2026-08-06 04:54:15 on dist-test-slave-1x4g"
server_key: "5bfb0ea02f69c61edbb5c9494689aefe"
server_key_iv: "cdbb02d07f4db337c4b5b5d545ee8831"
server_key_version: "kuduclusterkey@0"
I20260806 04:54:15.875221 26902 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 04:54:15.888154 26918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:15.889696 26902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:54:15.889904 26902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "4fbd57011b1a4b08938c4daef6bf1068"
format_stamp: "Formatted at 2026-08-06 04:54:15 on dist-test-slave-1x4g"
server_key: "5bfb0ea02f69c61edbb5c9494689aefe"
server_key_iv: "cdbb02d07f4db337c4b5b5d545ee8831"
server_key_version: "kuduclusterkey@0"
I20260806 04:54:15.890136 26902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:15.902730 26902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:15.906486 26902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:15.906807 26902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:15.908036 26902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:15.909535 26902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:54:15.909641 26902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:15.909754 26902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:54:15.909829 26902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:15.951655 26902 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:43497
I20260806 04:54:15.951674 27032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:43497 every 8 connection(s)
I20260806 04:54:15.952762 26902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:54:15.954296 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 26902
I20260806 04:54:15.954502 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:54:15.955133 13906 external_mini_cluster.cc:1428] Setting key 71d1248a0543ec34f19fe3636ca384d4
Aug 06 04:54:15 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992055, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:15.982260 27035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:15.960951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:34501 (local address 127.13.148.190:37709)
0806 04:54:15.964169 (+ 3218us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:15.964174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:15.967192 (+ 3018us) server_negotiation.cc:407] Connection header received
0806 04:54:15.967236 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:15.967238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:15.967305 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:15.967386 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:15.969364 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:15.970330 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:15.972547 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:15.972987 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:15.973092 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:15.976614 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:15.976619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:15.976622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:15.976676 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:15.979282 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:15.979975 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:15.979980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:15.979986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:15.980159 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:15.980589 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:15.980594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:15.980598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:15.980943 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:15.981127 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:15.981691 (+ 564us) server_negotiation.cc:299] Negotiation successful
0806 04:54:15.981926 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3109,"thread_start_us":3009,"threads_started":1}
I20260806 04:54:15.983312 27036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:15.961837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:37709 (local address 127.13.148.131:34501)
0806 04:54:15.965417 (+ 3580us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:15.965498 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:15.966962 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:15.967678 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:15.967689 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:15.968249 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:15.969196 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:15.969221 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:15.970505 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:15.970511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:15.972376 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:15.972390 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:15.973213 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:15.973220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:15.973498 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:15.974166 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:15.974192 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:15.976417 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:15.979432 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:15.979440 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:15.979453 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:15.979834 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:15.980269 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:15.980274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:15.980278 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:15.980474 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:15.981090 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:15.981097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:15.981390 (+ 293us) client_negotiation.cc:769] Sending connection context
0806 04:54:15.981623 (+ 233us) client_negotiation.cc:240] Negotiation successful
0806 04:54:15.982092 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3278,"thread_start_us":3025,"threads_started":1}
I20260806 04:54:15.984551 27033 heartbeater.cc:347] Connected to a master server at 127.13.148.190:37709
I20260806 04:54:15.984890 27033 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:15.985567 27033 heartbeater.cc:511] Master 127.13.148.190:37709 requested a full tablet report, sending...
I20260806 04:54:15.986822 26536 ts_manager.cc:194] Registered new tserver with Master: 4fbd57011b1a4b08938c4daef6bf1068 (127.13.148.131:43497)
I20260806 04:54:15.987577 26536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:34501
I20260806 04:54:15.995393 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:54:15.997778 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 26617
Aug 06 04:54:16 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:54:16.015708 27038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:15.998793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.13.148.129:41749)
0806 04:54:15.999508 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:15.999514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:15.999536 (+ 22us) server_negotiation.cc:407] Connection header received
0806 04:54:15.999588 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:15.999593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:15.999757 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:15.999887 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:16.001973 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:16.003124 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:16.004933 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:16.005456 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:16.005546 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:16.010274 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:16.010285 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:16.010298 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:16.010330 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:16.013147 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:16.013756 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:16.013761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:16.013763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:16.013893 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:16.014235 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:16.014239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:16.014245 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:16.014576 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:16.014858 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:16.015111 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 04:54:16.015359 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":381,"threads_started":1}
I20260806 04:54:16.017028 26747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:16.090634 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 26617
I20260806 04:54:16.107131 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 26758
Aug 06 04:54:16 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:54:16.121838 27040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:16.108063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40234 (local address 127.13.148.130:41207)
0806 04:54:16.108631 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:16.108638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:16.108663 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:54:16.108722 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:16.108728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:16.108990 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:16.109191 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:16.110442 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:16.111410 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:16.112715 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:16.113290 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:16.113377 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:16.117099 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:16.117109 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:16.117122 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:16.117161 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:16.119479 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:16.120032 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:16.120038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:16.120041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:16.120164 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:16.120462 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:16.120465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:16.120467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:16.120770 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:16.121086 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:16.121264 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 04:54:16.121468 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":291,"threads_started":1}
I20260806 04:54:16.123001 26891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:16.194628 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 26758
I20260806 04:54:16.211822 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 26902
Aug 06 04:54:16 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
I20260806 04:54:16.226321 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:16.212657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.13.148.131:43497)
0806 04:54:16.213261 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:16.213267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:16.213292 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:54:16.213343 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:16.213349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:16.213532 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:16.213769 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:16.214893 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:16.215894 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:16.217274 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:16.217800 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:16.217962 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:16.221956 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:16.221966 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:16.221977 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:16.222007 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:16.224234 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:16.224860 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:16.224865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:16.224867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:16.224982 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:16.225263 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:16.225266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:16.225268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:16.225548 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:16.225752 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:16.225910 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 04:54:16.226074 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":333,"threads_started":1}
I20260806 04:54:16.227399 27028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:16.301749 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 26902
I20260806 04:54:16.318531 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 26504
I20260806 04:54:16.319614 26567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:16.416987 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 26504
I20260806 04:54:16.440203 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:37709
--webserver_interface=127.13.148.190
--webserver_port=43683
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:37709
--ranger_config_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.148:59515/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.148.148:59515/kms with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:54:16.742559 27045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:16.742839 27045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:16.742945 27045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:16.749665 27045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:54:16.749761 27045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:16.749814 27045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:16.749872 27045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:54:16.749922 27045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:54:16.759505 27045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.148.148:59515/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:37709
--ranger_config_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:37709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=43683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:16.760627 27045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:16.761835 27045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:16.773289 27052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:16.773404 27051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:16.773413 27054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:16.774369 27045 server_base.cc:1034] running on GCE node
I20260806 04:54:16.775177 27045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:16.776626 27045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:16.777865 27045 hybrid_clock.cc:648] HybridClock initialized: now 1785992056777829 us; error 81 us; skew 500 ppm
I20260806 04:54:16.778223 27045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:16 dist-test-slave-1x4g krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992056, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:16.781481 27045 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:54:16.782924 27045 webserver.cc:460] Webserver started at http://127.13.148.190:43683/ using document root <none> and password file <none>
I20260806 04:54:16.783557 27045 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:16.783704 27045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 04:54:16 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992056, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
I20260806 04:54:16.836109 27045 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 04:54:16.839445 27062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:16.840981 27045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:54:16.841226 27045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "f183e8427aa34d1a812331a828cdb147"
format_stamp: "Formatted at 2026-08-06 04:54:12 on dist-test-slave-1x4g"
server_key: "0ffba0f54cbca6da0e82084277b96c05"
server_key_iv: "95c9fc4765f45e9814c5ef0997344962"
server_key_version: "kuduclusterkey@0"
I20260806 04:54:16.842168 27045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:16.859640 27045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:16.863121 27045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:16.863394 27045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:16.881614 27045 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:37709
I20260806 04:54:16.881633 27114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:37709 every 8 connection(s)
I20260806 04:54:16.883190 27045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:54:16.889891 27115 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:54:16.893079 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 27045
I20260806 04:54:16.893144 27115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147: Bootstrap starting.
I20260806 04:54:16.894088 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.148:59515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:41749
--local_ip_for_outbound_sockets=127.13.148.129
--tserver_master_addrs=127.13.148.190:37709
--webserver_port=42603
--webserver_interface=127.13.148.129
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.148.148:59515/kms with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:54:16.913853 27115 log.cc:824] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:16.922559 27115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:54:16.923043 27115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147: Bootstrap complete.
I20260806 04:54:16.931798 27115 raft_consensus.cc:348] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183e8427aa34d1a812331a828cdb147" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37709 } }
I20260806 04:54:16.932622 27115 raft_consensus.cc:729] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f183e8427aa34d1a812331a828cdb147, State: Initialized, Role: FOLLOWER
I20260806 04:54:16.933254 27115 consensus_queue.cc:260] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183e8427aa34d1a812331a828cdb147" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37709 } }
I20260806 04:54:16.933485 27115 raft_consensus.cc:388] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:54:16.933624 27115 raft_consensus.cc:482] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:54:16.933812 27115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:54:16.937635 27115 raft_consensus.cc:504] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183e8427aa34d1a812331a828cdb147" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37709 } }
I20260806 04:54:16.938107 27115 leader_election.cc:302] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f183e8427aa34d1a812331a828cdb147; no voters:
I20260806 04:54:16.938727 27115 leader_election.cc:288] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 04:54:16.938862 27119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:54:16.940065 27119 raft_consensus.cc:686] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [term 2 LEADER]: Becoming Leader. State: Replica: f183e8427aa34d1a812331a828cdb147, State: Running, Role: LEADER
I20260806 04:54:16.940670 27119 consensus_queue.cc:237] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183e8427aa34d1a812331a828cdb147" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37709 } }
I20260806 04:54:16.940939 27115 sys_catalog.cc:564] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:54:16.943222 27121 sys_catalog.cc:455] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f183e8427aa34d1a812331a828cdb147. Latest consensus state: current_term: 2 leader_uuid: "f183e8427aa34d1a812331a828cdb147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183e8427aa34d1a812331a828cdb147" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37709 } } }
I20260806 04:54:16.943617 27121 sys_catalog.cc:458] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [sys.catalog]: This master's current role is: LEADER
I20260806 04:54:16.944242 27120 sys_catalog.cc:455] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f183e8427aa34d1a812331a828cdb147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183e8427aa34d1a812331a828cdb147" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37709 } } }
I20260806 04:54:16.944528 27120 sys_catalog.cc:458] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147 [sys.catalog]: This master's current role is: LEADER
I20260806 04:54:16.955008 27125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:54:16.960928 27125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:54:16.961746 27115 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 04:54:16.962363 27125 catalog_manager.cc:1259] Loaded cluster ID: 2427dc1163e04d0993c668bf4279c225
I20260806 04:54:16.962445 27125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:54:16.966665 27125 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:54:16.968856 27125 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147: Loaded TSK: 0
I20260806 04:54:16.970412 27125 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 04:54:17.310420 27117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:17.310750 27117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:17.310937 27117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:17.321820 27117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:17.322029 27117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:17.322273 27117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:54:17.337821 27117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.148.148:59515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:41749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=42603
--tserver_master_addrs=127.13.148.190:37709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:17.339375 27117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:17.341396 27117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:17.382323 27153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:17.408121 27155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:17.412452 27158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:17.413084 27117 server_base.cc:1034] running on GCE node
I20260806 04:54:17.413942 27117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:17.421067 27117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:17.428588 27117 hybrid_clock.cc:648] HybridClock initialized: now 1785992057428499 us; error 81 us; skew 500 ppm
I20260806 04:54:17.429040 27117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:17 dist-test-slave-1x4g krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992057, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:17.433476 27117 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:54:17.438979 27117 webserver.cc:460] Webserver started at http://127.13.148.129:42603/ using document root <none> and password file <none>
I20260806 04:54:17.439847 27117 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:17.440023 27117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 04:54:17 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992057, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
I20260806 04:54:17.487891 27117 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.003s sys 0.008s
I20260806 04:54:17.498924 27165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:17.500736 27117 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.004s
I20260806 04:54:17.500975 27117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "a6d88d96cfa84d2b9482bac72f5cbdcc"
format_stamp: "Formatted at 2026-08-06 04:54:14 on dist-test-slave-1x4g"
server_key: "c0f024473464cefda23ad86c7604e6d3"
server_key_iv: "a08f3841d5c0681b8a3af859150b9708"
server_key_version: "kuduclusterkey@0"
I20260806 04:54:17.502547 27117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:17.539664 27117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:17.544698 27117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:17.545023 27117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:17.550767 27117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:17.552524 27117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:54:17.552645 27117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:17.552760 27117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:54:17.552881 27117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 04:54:17.612277 27117 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:41749
I20260806 04:54:17.612286 27278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:41749 every 8 connection(s)
I20260806 04:54:17.614140 27117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:54:17.622023 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 27117
I20260806 04:54:17.622846 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.148:59515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:41207
--local_ip_for_outbound_sockets=127.13.148.130
--tserver_master_addrs=127.13.148.190:37709
--webserver_port=35981
--webserver_interface=127.13.148.130
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.148.148:59515/kms with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:54:17 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992057, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:17.643749 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:17.617235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:37709 (local address 127.13.148.129:35335)
0806 04:54:17.618093 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:17.618148 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:17.619935 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:17.621512 (+ 1577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:17.621522 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:17.622117 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:17.622966 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:17.622994 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:17.625356 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:17.625362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:17.628295 (+ 2933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:17.628312 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:17.630112 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:17.630120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:17.630390 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:17.631209 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:17.631236 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:17.633847 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:17.638402 (+ 4555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:17.638411 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:17.638445 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:17.639034 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:17.639589 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:17.639594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:17.639598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:17.639869 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:17.641058 (+ 1189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:17.641069 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:17.641490 (+ 421us) client_negotiation.cc:769] Sending connection context
0806 04:54:17.641769 (+ 279us) client_negotiation.cc:240] Negotiation successful
0806 04:54:17.642350 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":362,"threads_started":1}
I20260806 04:54:17.646458 27282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:17.617311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:35335 (local address 127.13.148.190:37709)
0806 04:54:17.618872 (+ 1561us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:17.618889 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:17.618949 (+ 60us) server_negotiation.cc:407] Connection header received
0806 04:54:17.620517 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:17.620529 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:17.620916 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:17.621336 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:17.623208 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:17.625142 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:17.628481 (+ 3339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:17.629512 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:17.629702 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:17.634170 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:17.634182 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:17.634201 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:17.634247 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:17.638223 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:17.639205 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:17.639216 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:17.639226 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:17.639429 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:17.640028 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:17.640035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:17.640041 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:17.640735 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:17.640951 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:17.644650 (+ 3699us) server_negotiation.cc:299] Negotiation successful
0806 04:54:17.645146 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1324,"thread_start_us":1121,"threads_started":1}
I20260806 04:54:17.649857 27279 heartbeater.cc:347] Connected to a master server at 127.13.148.190:37709
I20260806 04:54:17.650195 27279 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:17.651199 27279 heartbeater.cc:511] Master 127.13.148.190:37709 requested a full tablet report, sending...
W20260806 04:54:17.654533 27279 heartbeater.cc:644] Failed to heartbeat to 127.13.148.190:37709 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 04:54:17.654747 27279 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:17.655005 27279 heartbeater.cc:511] Master 127.13.148.190:37709 requested a full tablet report, sending...
I20260806 04:54:17.656066 27279 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:17.656283 27279 heartbeater.cc:511] Master 127.13.148.190:37709 requested a full tablet report, sending...
W20260806 04:54:17.657267 27279 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 04:54:17.987612 27283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:17.987887 27283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:17.988076 27283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:17.995074 27283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:17.995203 27283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:17.995350 27283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:54:18.005136 27283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.148.148:59515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:41207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=35981
--tserver_master_addrs=127.13.148.190:37709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:18.006569 27283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:18.008188 27283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:18.022119 27297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:18.022189 27299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:18.027379 27301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:18.028119 27283 server_base.cc:1034] running on GCE node
I20260806 04:54:18.028734 27283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:18.029695 27283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:18.030901 27283 hybrid_clock.cc:648] HybridClock initialized: now 1785992058030852 us; error 39 us; skew 500 ppm
I20260806 04:54:18.031234 27283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:18 dist-test-slave-1x4g krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992058, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:18.037405 27283 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:54:18.039434 27283 webserver.cc:460] Webserver started at http://127.13.148.130:35981/ using document root <none> and password file <none>
I20260806 04:54:18.040431 27283 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:18.040618 27283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 04:54:18 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992058, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
I20260806 04:54:18.062512 27283 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 04:54:18.069447 27309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:18.071322 27283 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.004s
I20260806 04:54:18.071487 27283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "44cff7e36fef48c2b60bbde0dab7f926"
format_stamp: "Formatted at 2026-08-06 04:54:15 on dist-test-slave-1x4g"
server_key: "196f5dff746ffecd9b99927465192c53"
server_key_iv: "a289e61d19f43195512e28d31ee20093"
server_key_version: "kuduclusterkey@0"
I20260806 04:54:18.072916 27283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:18.125212 27283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:18.128978 27283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:18.129232 27283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:18.130360 27283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:18.131932 27283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:54:18.132052 27283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:18.132182 27283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:54:18.132274 27283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:54:18 dist-test-slave-1x4g krb5kdc[26051](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785992058, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:18.182164 27283 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:41207
I20260806 04:54:18.182221 27423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:41207 every 8 connection(s)
I20260806 04:54:18.183528 27283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:54:18.185863 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 27283
I20260806 04:54:18.186534 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.148:59515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:43497
--local_ip_for_outbound_sockets=127.13.148.131
--tserver_master_addrs=127.13.148.190:37709
--webserver_port=41835
--webserver_interface=127.13.148.131
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.148.148:59515/kms with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:54:18 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992058, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:18.219893 27427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:18.187718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:45765 (local address 127.13.148.190:37709)
0806 04:54:18.197195 (+ 9477us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:18.197201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:18.197225 (+ 24us) server_negotiation.cc:407] Connection header received
0806 04:54:18.199463 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:18.199468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:18.199558 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:18.199665 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:18.201902 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:18.203013 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:18.205377 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:18.206062 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:18.206160 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:18.210806 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:18.210815 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:18.210823 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:18.210895 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:18.214863 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:18.216726 (+ 1863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:18.216734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:18.216741 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:18.216927 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:18.217551 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:18.217558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:18.217564 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:18.217986 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:18.218159 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:18.219207 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0806 04:54:18.219473 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9341,"thread_start_us":9208,"threads_started":1}
I20260806 04:54:18.222436 27426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:18.187124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:37709 (local address 127.13.148.130:45765)
0806 04:54:18.195998 (+ 8874us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:18.196074 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:18.199193 (+ 3119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:18.200166 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:18.200181 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:18.200866 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:18.201723 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:18.201748 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:18.203197 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:18.203203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:18.205236 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:18.205252 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:18.206502 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:18.206510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:18.206814 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:18.207606 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:18.207645 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:18.210565 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:18.215021 (+ 4456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:18.215032 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:18.215051 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:18.215707 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:18.217094 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:18.217101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:18.217108 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:18.217391 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:18.218262 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:18.218272 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:18.218736 (+ 464us) client_negotiation.cc:769] Sending connection context
0806 04:54:18.220512 (+ 1776us) client_negotiation.cc:240] Negotiation successful
0806 04:54:18.221159 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8553,"thread_start_us":8340,"threads_started":1}
I20260806 04:54:18.224191 27424 heartbeater.cc:347] Connected to a master server at 127.13.148.190:37709
I20260806 04:54:18.224546 27424 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:18.225419 27424 heartbeater.cc:511] Master 127.13.148.190:37709 requested a full tablet report, sending...
I20260806 04:54:18.227337 27079 ts_manager.cc:194] Registered new tserver with Master: 44cff7e36fef48c2b60bbde0dab7f926 (127.13.148.130:41207)
I20260806 04:54:18.229301 27079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:45765
W20260806 04:54:18.563179 27428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:18.563439 27428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:18.563584 27428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:18.570869 27428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:18.571048 27428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:18.571249 27428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:54:18.582052 27428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.148.148:59515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:43497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=41835
--tserver_master_addrs=127.13.148.190:37709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:18.583362 27428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:18.584648 27428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 06 04:54:18 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992058, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
W20260806 04:54:18.619338 27434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:18.621434 27436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:18.624202 27428 server_base.cc:1034] running on GCE node
W20260806 04:54:18.624698 27438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:18.625557 27428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:18.636528 27428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:18.637897 27428 hybrid_clock.cc:648] HybridClock initialized: now 1785992058637810 us; error 49 us; skew 500 ppm
I20260806 04:54:18.638280 27428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:18 dist-test-slave-1x4g krb5kdc[26051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992058, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:18.642524 27428 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:54:18.657483 27279 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:18.658051 27279 heartbeater.cc:511] Master 127.13.148.190:37709 requested a full tablet report, sending...
I20260806 04:54:18.659706 27079 ts_manager.cc:194] Registered new tserver with Master: a6d88d96cfa84d2b9482bac72f5cbdcc (127.13.148.129:41749)
I20260806 04:54:18.661288 27079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:35335
I20260806 04:54:18.689169 27428 webserver.cc:460] Webserver started at http://127.13.148.131:41835/ using document root <none> and password file <none>
I20260806 04:54:18.689946 27428 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:18.690141 27428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 04:54:18 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992058, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
I20260806 04:54:18.710223 27428 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 04:54:18.713297 27446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:18.714622 27428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 04:54:18.714783 27428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "4fbd57011b1a4b08938c4daef6bf1068"
format_stamp: "Formatted at 2026-08-06 04:54:15 on dist-test-slave-1x4g"
server_key: "5bfb0ea02f69c61edbb5c9494689aefe"
server_key_iv: "cdbb02d07f4db337c4b5b5d545ee8831"
server_key_version: "kuduclusterkey@0"
I20260806 04:54:18.715648 27428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:18.739317 27428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:18.743328 27428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:18.743592 27428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:18.745267 27428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:18.746810 27428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:54:18.746920 27428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:18.747049 27428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:54:18.747130 27428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:18.842046 27428 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:43497
I20260806 04:54:18.842124 27561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:43497 every 8 connection(s)
I20260806 04:54:18.843624 27428 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:54:18.849212 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 27428
Aug 06 04:54:18 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992058, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:18.891047 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:18.850218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.13.148.190:37709)
0806 04:54:18.865555 (+ 15337us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:18.865563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:18.865584 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:54:18.865660 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:18.865667 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:18.865765 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:18.865885 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:18.870372 (+ 4487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:18.871842 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:18.874650 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:18.875226 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:18.875341 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:18.878721 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:18.878727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:18.878731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:18.878764 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:18.882246 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:18.886098 (+ 3852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:18.886102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:18.886106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:18.886248 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:18.889391 (+ 3143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:18.889395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:18.889398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:18.889787 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:18.890016 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:18.890451 (+ 435us) server_negotiation.cc:299] Negotiation successful
0806 04:54:18.890671 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15176,"thread_start_us":17161,"threads_started":1}
I20260806 04:54:18.893750 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:18.857051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:45993 (local address 127.13.148.190:37709)
0806 04:54:18.866388 (+ 9337us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:18.866395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:18.867368 (+ 973us) server_negotiation.cc:407] Connection header received
0806 04:54:18.872069 (+ 4701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:18.872076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:18.872169 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:18.872246 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:18.875565 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:18.876514 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:18.879484 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:18.880143 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:18.880252 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:18.886171 (+ 5919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:18.886179 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:18.886186 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:18.886213 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:18.888533 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:18.889379 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:18.889384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:18.889386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:18.889569 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:18.890089 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:18.890094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:18.890097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:18.890430 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:18.890557 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:18.893252 (+ 2695us) server_negotiation.cc:299] Negotiation successful
0806 04:54:18.893427 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9208,"spinlock_wait_cycles":12160,"thread_start_us":9179,"threads_started":1}
I20260806 04:54:18.896492 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:18.858270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:37709 (local address 127.13.148.131:45993)
0806 04:54:18.867025 (+ 8755us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:18.867079 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:18.870001 (+ 2922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:18.872897 (+ 2896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:18.872911 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:18.873638 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:18.874463 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:18.874486 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:18.876729 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:18.876734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:18.879333 (+ 2599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:18.879356 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:18.880654 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:18.880663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:18.881011 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:18.881688 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:18.881719 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:18.885936 (+ 4217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:18.888651 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:18.888661 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:18.888678 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:18.889270 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:18.889697 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:18.889702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:18.889705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:18.889940 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:18.890656 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:18.890665 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:18.892581 (+ 1916us) client_negotiation.cc:769] Sending connection context
0806 04:54:18.894767 (+ 2186us) client_negotiation.cc:240] Negotiation successful
0806 04:54:18.895251 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8419,"thread_start_us":8168,"threads_started":1}
I20260806 04:54:18.898500 27562 heartbeater.cc:347] Connected to a master server at 127.13.148.190:37709
I20260806 04:54:18.898821 27562 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:18.899571 27562 heartbeater.cc:511] Master 127.13.148.190:37709 requested a full tablet report, sending...
I20260806 04:54:18.901196 27079 ts_manager.cc:194] Registered new tserver with Master: 4fbd57011b1a4b08938c4daef6bf1068 (127.13.148.131:43497)
I20260806 04:54:18.902189 27079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:45993
I20260806 04:54:18.906767 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:54:18.932284 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:18.916249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53840 (local address 127.13.148.190:37709)
0806 04:54:18.916385 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:18.916389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:18.916983 (+ 594us) server_negotiation.cc:407] Connection header received
0806 04:54:18.917137 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:18.917142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:18.917236 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:18.917342 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:18.918950 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:18.919826 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:18.921645 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:18.922154 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:18.922232 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:18.925811 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:18.925818 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:18.925822 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:18.925858 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:18.929089 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:18.929952 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:18.929957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:18.929961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:18.930125 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:18.930608 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:18.930613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:18.930617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:18.931035 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:18.931337 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:18.931678 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 04:54:18.931896 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Aug 06 04:54:18 dist-test-slave-1x4g krb5kdc[26051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992058, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for HTTP/127.13.148.148@KRBTEST.COM
I20260806 04:54:18.939356 27079 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:54:18.949468 27079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:54:19.034580 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:18.987987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:43497 (local address 127.0.0.1:54412)
0806 04:54:19.010196 (+ 22209us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.010246 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:19.010460 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:19.012038 (+ 1578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:19.012049 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:19.012089 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:19.012634 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:19.012650 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.022666 (+ 10016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.022672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:19.031310 (+ 8638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:19.031323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.033559 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.033565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:19.033753 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:19.033803 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 04:54:19.033931 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 04:54:19.034132 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21941,"mutex_wait_us":1,"thread_start_us":31841,"threads_started":1}
I20260806 04:54:19.035550 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:18.988329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.13.148.131:43497)
0806 04:54:19.009618 (+ 21289us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:19.009630 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:19.010921 (+ 1291us) server_negotiation.cc:407] Connection header received
0806 04:54:19.011051 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:19.011062 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:19.011403 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:19.011613 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:19.020217 (+ 8604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.021885 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.031489 (+ 9604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.033021 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.033215 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:19.033400 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:19.034986 (+ 1586us) server_negotiation.cc:299] Negotiation successful
0806 04:54:19.035161 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21040,"thread_start_us":22318,"threads_started":1}
I20260806 04:54:19.040403 27495 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf52319410a4e3bab192a6b934e3eeb (DEFAULT_TABLE table=test-table [id=445c4b7774934790a116c76a29d3e329]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:54:19.041986 27495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf52319410a4e3bab192a6b934e3eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:19.045089 27578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:18.989816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:41749 (local address 127.0.0.1:58332)
0806 04:54:19.021175 (+ 31359us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.021194 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:19.021345 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:19.031516 (+ 10171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:19.031521 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:19.031543 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:19.031876 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:19.031902 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.040428 (+ 8526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.040437 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:19.042435 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:19.042449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.044283 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.044289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:19.044453 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:19.044484 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 04:54:19.044575 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 04:54:19.044695 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30153,"thread_start_us":54627,"threads_started":1}
I20260806 04:54:19.045967 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:18.991025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.13.148.130:41207)
0806 04:54:19.027602 (+ 36577us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:19.027610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:19.030680 (+ 3070us) server_negotiation.cc:407] Connection header received
0806 04:54:19.030778 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:19.030785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:19.030998 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:19.031192 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:19.034499 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.036163 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.038692 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.039972 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.040143 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:19.040298 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:19.045512 (+ 5214us) server_negotiation.cc:299] Negotiation successful
0806 04:54:19.045668 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36425,"thread_start_us":36598,"threads_started":1}
I20260806 04:54:19.047055 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:18.990647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.13.148.129:41749)
0806 04:54:19.029330 (+ 38683us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:19.029338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:19.029367 (+ 29us) server_negotiation.cc:407] Connection header received
0806 04:54:19.029466 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:19.029473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:19.029715 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:19.032079 (+ 2364us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:19.032305 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.033844 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.042774 (+ 8930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.044066 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.044254 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:19.044425 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:19.046589 (+ 2164us) server_negotiation.cc:299] Negotiation successful
0806 04:54:19.046737 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38494,"thread_start_us":39193,"threads_started":1}
I20260806 04:54:19.047457 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:18.990157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:41207 (local address 127.0.0.1:40238)
0806 04:54:19.030416 (+ 40259us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.030435 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:19.030549 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:19.033976 (+ 3427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:19.033980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:19.034005 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:19.034320 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:19.034331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.036554 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.036559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:19.038504 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:19.038519 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.044745 (+ 6226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.044751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:19.044961 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:19.044997 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 04:54:19.045070 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 04:54:19.046368 (+ 1298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40134,"thread_start_us":19641,"threads_started":1}
I20260806 04:54:19.050944 27359 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf52319410a4e3bab192a6b934e3eeb (DEFAULT_TABLE table=test-table [id=445c4b7774934790a116c76a29d3e329]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:54:19.052363 27359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf52319410a4e3bab192a6b934e3eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:19.066075 27584 tablet_bootstrap.cc:492] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068: Bootstrap starting.
I20260806 04:54:19.066398 27214 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf52319410a4e3bab192a6b934e3eeb (DEFAULT_TABLE table=test-table [id=445c4b7774934790a116c76a29d3e329]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:54:19.068204 27214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf52319410a4e3bab192a6b934e3eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:19.072796 27584 tablet_bootstrap.cc:654] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:19.074070 27584 log.cc:824] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:19.103356 27587 tablet_bootstrap.cc:492] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc: Bootstrap starting.
I20260806 04:54:19.103812 27584 tablet_bootstrap.cc:492] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068: No bootstrap required, opened a new log
I20260806 04:54:19.104120 27584 ts_tablet_manager.cc:1397] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.015s
I20260806 04:54:19.106251 27586 tablet_bootstrap.cc:492] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926: Bootstrap starting.
I20260806 04:54:19.109978 27587 tablet_bootstrap.cc:654] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:19.113997 27586 tablet_bootstrap.cc:654] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:19.114264 27587 log.cc:824] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:19.115010 27584 raft_consensus.cc:348] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fbd57011b1a4b08938c4daef6bf1068" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43497 } } peers { permanent_uuid: "a6d88d96cfa84d2b9482bac72f5cbdcc" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 41749 } } peers { permanent_uuid: "44cff7e36fef48c2b60bbde0dab7f926" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 41207 } }
I20260806 04:54:19.115520 27586 log.cc:824] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:19.115581 27584 raft_consensus.cc:374] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:19.115674 27584 raft_consensus.cc:729] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fbd57011b1a4b08938c4daef6bf1068, State: Initialized, Role: FOLLOWER
I20260806 04:54:19.116446 27584 consensus_queue.cc:260] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fbd57011b1a4b08938c4daef6bf1068" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43497 } } peers { permanent_uuid: "a6d88d96cfa84d2b9482bac72f5cbdcc" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 41749 } } peers { permanent_uuid: "44cff7e36fef48c2b60bbde0dab7f926" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 41207 } }
I20260806 04:54:19.126420 27584 ts_tablet_manager.cc:1428] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068: Time spent starting tablet: real 0.022s user 0.008s sys 0.013s
I20260806 04:54:19.126809 27562 heartbeater.cc:503] Master 127.13.148.190:37709 was elected leader, sending a full tablet report...
I20260806 04:54:19.127255 27586 tablet_bootstrap.cc:492] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926: No bootstrap required, opened a new log
I20260806 04:54:19.127504 27586 ts_tablet_manager.cc:1397] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.004s
I20260806 04:54:19.136554 27587 tablet_bootstrap.cc:492] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc: No bootstrap required, opened a new log
I20260806 04:54:19.136974 27587 ts_tablet_manager.cc:1397] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.003s
I20260806 04:54:19.138783 27586 raft_consensus.cc:348] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fbd57011b1a4b08938c4daef6bf1068" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43497 } } peers { permanent_uuid: "a6d88d96cfa84d2b9482bac72f5cbdcc" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 41749 } } peers { permanent_uuid: "44cff7e36fef48c2b60bbde0dab7f926" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 41207 } }
I20260806 04:54:19.139267 27586 raft_consensus.cc:374] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:19.139400 27586 raft_consensus.cc:729] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44cff7e36fef48c2b60bbde0dab7f926, State: Initialized, Role: FOLLOWER
I20260806 04:54:19.140187 27586 consensus_queue.cc:260] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fbd57011b1a4b08938c4daef6bf1068" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43497 } } peers { permanent_uuid: "a6d88d96cfa84d2b9482bac72f5cbdcc" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 41749 } } peers { permanent_uuid: "44cff7e36fef48c2b60bbde0dab7f926" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 41207 } }
I20260806 04:54:19.148488 27586 ts_tablet_manager.cc:1428] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926: Time spent starting tablet: real 0.021s user 0.006s sys 0.014s
I20260806 04:54:19.148888 27424 heartbeater.cc:503] Master 127.13.148.190:37709 was elected leader, sending a full tablet report...
I20260806 04:54:19.150059 27587 raft_consensus.cc:348] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fbd57011b1a4b08938c4daef6bf1068" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43497 } } peers { permanent_uuid: "a6d88d96cfa84d2b9482bac72f5cbdcc" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 41749 } } peers { permanent_uuid: "44cff7e36fef48c2b60bbde0dab7f926" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 41207 } }
I20260806 04:54:19.150580 27587 raft_consensus.cc:374] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:19.150722 27587 raft_consensus.cc:729] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d88d96cfa84d2b9482bac72f5cbdcc, State: Initialized, Role: FOLLOWER
I20260806 04:54:19.151434 27587 consensus_queue.cc:260] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fbd57011b1a4b08938c4daef6bf1068" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43497 } } peers { permanent_uuid: "a6d88d96cfa84d2b9482bac72f5cbdcc" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 41749 } } peers { permanent_uuid: "44cff7e36fef48c2b60bbde0dab7f926" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 41207 } }
I20260806 04:54:19.155010 27279 heartbeater.cc:503] Master 127.13.148.190:37709 was elected leader, sending a full tablet report...
I20260806 04:54:19.155669 27587 ts_tablet_manager.cc:1428] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc: Time spent starting tablet: real 0.019s user 0.015s sys 0.001s
I20260806 04:54:19.180922 27590 raft_consensus.cc:482] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:54:19.181237 27590 raft_consensus.cc:504] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fbd57011b1a4b08938c4daef6bf1068" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43497 } } peers { permanent_uuid: "a6d88d96cfa84d2b9482bac72f5cbdcc" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 41749 } } peers { permanent_uuid: "44cff7e36fef48c2b60bbde0dab7f926" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 41207 } }
I20260806 04:54:19.183159 27590 leader_election.cc:288] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6d88d96cfa84d2b9482bac72f5cbdcc (127.13.148.129:41749), 44cff7e36fef48c2b60bbde0dab7f926 (127.13.148.130:41207)
W20260806 04:54:19.185958 27425 tablet.cc:2369] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:54:19.192303 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.183864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:41749 (local address 127.13.148.131:47649)
0806 04:54:19.184685 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.184704 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:19.185406 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:19.185979 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:19.185985 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:19.186013 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:19.186202 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:19.186215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.187747 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.187752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:19.189639 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:19.189656 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.191481 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.191488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:19.191689 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:19.191726 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 04:54:19.191829 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 04:54:19.191979 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708}
I20260806 04:54:19.193084 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.185018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:47649 (local address 127.13.148.129:41749)
0806 04:54:19.185212 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:19.185218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:19.185240 (+ 22us) server_negotiation.cc:407] Connection header received
0806 04:54:19.185577 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:19.185583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:19.185672 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:19.185776 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:19.186379 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.187527 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.189834 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.191096 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.191265 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:19.191361 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:19.192698 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0806 04:54:19.192852 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:54:19.193926 27234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf52319410a4e3bab192a6b934e3eeb" candidate_uuid: "4fbd57011b1a4b08938c4daef6bf1068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d88d96cfa84d2b9482bac72f5cbdcc" is_pre_election: true
I20260806 04:54:19.194415 27234 raft_consensus.cc:2454] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fbd57011b1a4b08938c4daef6bf1068 in term 0.
I20260806 04:54:19.195258 27447 leader_election.cc:302] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fbd57011b1a4b08938c4daef6bf1068, a6d88d96cfa84d2b9482bac72f5cbdcc; no voters:
I20260806 04:54:19.195713 27590 raft_consensus.cc:2781] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:54:19.195838 27590 raft_consensus.cc:482] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:54:19.195923 27590 raft_consensus.cc:3037] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:19.199211 27590 raft_consensus.cc:504] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fbd57011b1a4b08938c4daef6bf1068" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43497 } } peers { permanent_uuid: "a6d88d96cfa84d2b9482bac72f5cbdcc" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 41749 } } peers { permanent_uuid: "44cff7e36fef48c2b60bbde0dab7f926" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 41207 } }
I20260806 04:54:19.200204 27234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf52319410a4e3bab192a6b934e3eeb" candidate_uuid: "4fbd57011b1a4b08938c4daef6bf1068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d88d96cfa84d2b9482bac72f5cbdcc"
I20260806 04:54:19.200393 27234 raft_consensus.cc:3037] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:19.203367 27234 raft_consensus.cc:2454] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fbd57011b1a4b08938c4daef6bf1068 in term 1.
I20260806 04:54:19.203790 27447 leader_election.cc:302] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fbd57011b1a4b08938c4daef6bf1068, a6d88d96cfa84d2b9482bac72f5cbdcc; no voters:
I20260806 04:54:19.204169 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.185296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:56871 (local address 127.13.148.130:41207)
0806 04:54:19.185480 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:19.185486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:19.192616 (+ 7130us) server_negotiation.cc:407] Connection header received
0806 04:54:19.196817 (+ 4201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:19.196823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:19.196925 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:19.197049 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:19.197688 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.198747 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.201817 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.203005 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.203160 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:19.203233 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:19.203739 (+ 506us) server_negotiation.cc:299] Negotiation successful
0806 04:54:19.203897 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:54:19.205017 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.184275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:41207 (local address 127.13.148.131:56871)
0806 04:54:19.192519 (+ 8244us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.192538 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:19.196626 (+ 4088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:19.197251 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:19.197256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:19.197282 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:19.197499 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:19.197512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.199791 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.199795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:19.201624 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:19.201640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.203318 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.203323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:19.203519 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:19.203562 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 04:54:19.204542 (+ 980us) client_negotiation.cc:240] Negotiation successful
0806 04:54:19.204679 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8154,"threads_started":1}
I20260806 04:54:19.205492 27590 leader_election.cc:288] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [CANDIDATE]: Term 1 election: Requested vote from peers a6d88d96cfa84d2b9482bac72f5cbdcc (127.13.148.129:41749), 44cff7e36fef48c2b60bbde0dab7f926 (127.13.148.130:41207)
I20260806 04:54:19.205749 27590 raft_consensus.cc:2781] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:54:19.205928 27590 raft_consensus.cc:686] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [term 1 LEADER]: Becoming Leader. State: Replica: 4fbd57011b1a4b08938c4daef6bf1068, State: Running, Role: LEADER
I20260806 04:54:19.206553 27590 consensus_queue.cc:237] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fbd57011b1a4b08938c4daef6bf1068" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43497 } } peers { permanent_uuid: "a6d88d96cfa84d2b9482bac72f5cbdcc" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 41749 } } peers { permanent_uuid: "44cff7e36fef48c2b60bbde0dab7f926" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 41207 } }
I20260806 04:54:19.216912 27378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf52319410a4e3bab192a6b934e3eeb" candidate_uuid: "4fbd57011b1a4b08938c4daef6bf1068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44cff7e36fef48c2b60bbde0dab7f926"
I20260806 04:54:19.216936 27379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf52319410a4e3bab192a6b934e3eeb" candidate_uuid: "4fbd57011b1a4b08938c4daef6bf1068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44cff7e36fef48c2b60bbde0dab7f926" is_pre_election: true
I20260806 04:54:19.217515 27378 raft_consensus.cc:3037] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:19.220283 27378 raft_consensus.cc:2454] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fbd57011b1a4b08938c4daef6bf1068 in term 1.
I20260806 04:54:19.221714 27079 catalog_manager.cc:5495] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fbd57011b1a4b08938c4daef6bf1068 (127.13.148.131). New cstate: current_term: 1 leader_uuid: "4fbd57011b1a4b08938c4daef6bf1068" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fbd57011b1a4b08938c4daef6bf1068" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 43497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6d88d96cfa84d2b9482bac72f5cbdcc" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 41749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44cff7e36fef48c2b60bbde0dab7f926" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 41207 } health_report { overall_health: UNKNOWN } } }
I20260806 04:54:19.268352 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.262910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.13.148.131:43497)
0806 04:54:19.263124 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:19.263129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:19.263150 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:54:19.263292 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:19.263297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:19.263393 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:19.263584 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:54:19.263940 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.264983 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.266226 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.266767 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.266892 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:19.267275 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:54:19.267869 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:54:19.268012 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:19.268054 (+ 42us) server_negotiation.cc:299] Negotiation successful
0806 04:54:19.268124 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 04:54:19.275679 27378 raft_consensus.cc:1261] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926 [term 1 FOLLOWER]: Refusing update from remote peer 4fbd57011b1a4b08938c4daef6bf1068: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:54:19.275867 27234 raft_consensus.cc:1261] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc [term 1 FOLLOWER]: Refusing update from remote peer 4fbd57011b1a4b08938c4daef6bf1068: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:54:19.276489 27595 consensus_queue.cc:1035] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44cff7e36fef48c2b60bbde0dab7f926" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 41207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:54:19.276895 27590 consensus_queue.cc:1035] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6d88d96cfa84d2b9482bac72f5cbdcc" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 41749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:54:19.289487 27604 mvcc.cc:205] Tried to move back new op lower bound from 7315423474784636928 to 7315423474516094976. Current Snapshot: MvccSnapshot[applied={T|T < 7315423474784636928}]
I20260806 04:54:19.290699 27605 mvcc.cc:205] Tried to move back new op lower bound from 7315423474784636928 to 7315423474516094976. Current Snapshot: MvccSnapshot[applied={T|T < 7315423474784636928}]
I20260806 04:54:19.304709 27078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:54:19.305022 27078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:54:19.309191 27078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f183e8427aa34d1a812331a828cdb147: Sending DeleteTablet for 3 replicas of tablet 7bf52319410a4e3bab192a6b934e3eeb
I20260806 04:54:19.310381 27359 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bf52319410a4e3bab192a6b934e3eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:54:19 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:40238
I20260806 04:54:19.310453 27214 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bf52319410a4e3bab192a6b934e3eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:54:19 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:58332
I20260806 04:54:19.311017 27213 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bf52319410a4e3bab192a6b934e3eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:54:19 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:58332
I20260806 04:54:19.311589 27495 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bf52319410a4e3bab192a6b934e3eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:54:19 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:54412
I20260806 04:54:19.315512 27613 tablet_replica.cc:330] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926: stopping tablet replica
I20260806 04:54:19.315949 27613 raft_consensus.cc:2229] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:54:19.315994 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 27117
I20260806 04:54:19.318821 27615 tablet_replica.cc:330] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068: stopping tablet replica
I20260806 04:54:19.319259 27615 raft_consensus.cc:2229] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:54:19.319914 27613 raft_consensus.cc:2258] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:54:19.320643 27615 raft_consensus.cc:2258] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:54:19.321700 27614 tablet_replica.cc:330] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc: stopping tablet replica
I20260806 04:54:19.321976 27614 raft_consensus.cc:2229] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:54:19.322359 27614 raft_consensus.cc:2258] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:54:19.323777 27613 ts_tablet_manager.cc:1905] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:54:19.328059 27615 ts_tablet_manager.cc:1905] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:54:19.334522 27615 ts_tablet_manager.cc:1918] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:54:19.334736 27615 log.cc:1196] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/7bf52319410a4e3bab192a6b934e3eeb
I20260806 04:54:19.335374 27615 ts_tablet_manager.cc:1939] T 7bf52319410a4e3bab192a6b934e3eeb P 4fbd57011b1a4b08938c4daef6bf1068: Deleting consensus metadata
I20260806 04:54:19.337024 27065 catalog_manager.cc:4841] TS 4fbd57011b1a4b08938c4daef6bf1068 (127.13.148.131:43497): tablet 7bf52319410a4e3bab192a6b934e3eeb (table test-table [id=445c4b7774934790a116c76a29d3e329]) successfully deleted
I20260806 04:54:19.337364 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.316657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.13.148.129:41749)
0806 04:54:19.316840 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:19.316845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:19.316955 (+ 110us) server_negotiation.cc:407] Connection header received
0806 04:54:19.317132 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:19.317136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:19.317218 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:19.317318 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:19.322505 (+ 5187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.323483 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.325878 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.326390 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.326529 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:19.329626 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:19.329639 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:19.329655 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:19.329697 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:19.333569 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:19.334337 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:19.334346 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:19.334351 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:19.334515 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:19.334961 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:19.334966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:19.334969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:19.335373 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:19.335643 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:19.336739 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0806 04:54:19.336976 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:54:19.337975 27614 ts_tablet_manager.cc:1905] T 7bf52319410a4e3bab192a6b934e3eeb P a6d88d96cfa84d2b9482bac72f5cbdcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:54:19.338280 27274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:19.338621 27613 ts_tablet_manager.cc:1918] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:54:19.338884 27613 log.cc:1196] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/7bf52319410a4e3bab192a6b934e3eeb
I20260806 04:54:19.339426 27613 ts_tablet_manager.cc:1939] T 7bf52319410a4e3bab192a6b934e3eeb P 44cff7e36fef48c2b60bbde0dab7f926: Deleting consensus metadata
I20260806 04:54:19.341665 27064 catalog_manager.cc:4841] TS 44cff7e36fef48c2b60bbde0dab7f926 (127.13.148.130:41207): tablet 7bf52319410a4e3bab192a6b934e3eeb (table test-table [id=445c4b7774934790a116c76a29d3e329]) successfully deleted
I20260806 04:54:19.475818 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 27117
W20260806 04:54:19.502393 27063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.129:41749 (error 108)
I20260806 04:54:19.503363 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 27283
I20260806 04:54:19.518638 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.518225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:41749 (local address 127.0.0.1:36840)
0806 04:54:19.518365 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.518482 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 04:54:19.518956 27063 catalog_manager.cc:4584] TS a6d88d96cfa84d2b9482bac72f5cbdcc (127.13.148.129:41749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7bf52319410a4e3bab192a6b934e3eeb: Network error: Client connection negotiation failed: client connection to 127.13.148.129:41749: connect: Connection refused (error 111)
I20260806 04:54:19.519842 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.519541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:41749 (local address 127.0.0.1:36848)
0806 04:54:19.519648 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.519718 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 04:54:19.521456 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.504786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.13.148.130:41207)
0806 04:54:19.504994 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:19.505000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:19.505016 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:54:19.505075 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:19.505079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:19.505146 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:19.505229 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:19.506870 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.507762 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.509406 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.510022 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.510167 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:19.513641 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:19.513658 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:19.513677 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:19.513718 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:19.517472 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:19.518301 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:19.518310 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:19.518313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:19.518471 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:19.519259 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:19.519266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:19.519269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:19.519696 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:19.520185 (+ 489us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:19.520756 (+ 571us) server_negotiation.cc:299] Negotiation successful
0806 04:54:19.521099 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:54:19.522439 27419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:19.564589 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.564062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:41749 (local address 127.0.0.1:36852)
0806 04:54:19.564292 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.564393 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 04:54:19.565696 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.565329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:41749 (local address 127.0.0.1:36854)
0806 04:54:19.565478 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.565560 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 04:54:19.572923 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.572453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:41749 (local address 127.0.0.1:36860)
0806 04:54:19.572639 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.572721 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 04:54:19.574024 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.573645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:41749 (local address 127.0.0.1:36866)
0806 04:54:19.573794 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.573874 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 04:54:19.610205 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 27283
I20260806 04:54:19.614319 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.613879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:41749 (local address 127.0.0.1:36880)
0806 04:54:19.614107 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.614194 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260806 04:54:19.615278 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.614942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:41749 (local address 127.0.0.1:36890)
0806 04:54:19.615088 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.615162 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 04:54:19.634832 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 27428
I20260806 04:54:19.650795 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.650377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:41749 (local address 127.0.0.1:36898)
0806 04:54:19.650544 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.650634 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 04:54:19.650887 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.635579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.13.148.131:43497)
0806 04:54:19.635755 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:19.635759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:19.635780 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:54:19.636008 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:19.636011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:19.636073 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:19.636146 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:19.638062 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.638832 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.640875 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:19.641409 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:19.641504 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:19.644720 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:19.644731 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:19.644743 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:19.644835 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:19.647708 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:19.648482 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:19.648488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:19.648490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:19.648600 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:19.649251 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:19.649254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:19.649256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:19.649609 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:19.649841 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:19.650273 (+ 432us) server_negotiation.cc:299] Negotiation successful
0806 04:54:19.650512 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:54:19.651571 27557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:19.652187 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.651812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:41749 (local address 127.0.0.1:36906)
0806 04:54:19.651938 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.652021 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 04:54:19.715451 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.715032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:41749 (local address 127.0.0.1:36910)
0806 04:54:19.715227 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.715306 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 04:54:19.716507 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.716147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:41749 (local address 127.0.0.1:36914)
0806 04:54:19.716314 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.716384 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 04:54:19.754025 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.753597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:41749 (local address 127.0.0.1:36918)
0806 04:54:19.753766 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.753881 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 04:54:19.755051 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:19.754691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:41749 (local address 127.0.0.1:36930)
0806 04:54:19.754847 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:19.754929 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 04:54:19.758054 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 27428
I20260806 04:54:19.778047 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 27045
I20260806 04:54:19.779095 27110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:19.880929 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 27045
I20260806 04:54:19.903252 13906 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 04:54:20.238826 13906 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 04:54:20.239068 13906 mini_ranger.cc:66] Stopping Ranger...
I20260806 04:54:20.584208 13906 mini_ranger.cc:68] Stopped Ranger
2026-08-06 04:54:20.584 UTC [26081] LOG: received smart shutdown request
2026-08-06 04:54:20.587 UTC [26081] LOG: background worker "logical replication launcher" (PID 26090) exited with exit code 1
2026-08-06 04:54:20.587 UTC [26085] LOG: shutting down
2026-08-06 04:54:20.735 UTC [26081] LOG: database system is shut down
2026-08-06T04:54:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49126 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:54:20 dist-test-slave-1x4g krb5kdc[27629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:54:20 dist-test-slave-1x4g krb5kdc[27629](info): set up 2 sockets
Aug 06 04:54:20 dist-test-slave-1x4g krb5kdc[27629](info): commencing operation
krb5kdc: starting...
W20260806 04:54:21.853201 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:54:21 dist-test-slave-1x4g krb5kdc[27629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:54:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:54:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:54:22.051934 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:40147
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:40147,127.13.148.189:43297,127.13.148.188:40779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:54:22.351630 27643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:22.351881 27643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:22.351992 27643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:22.358482 27643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:54:22.358568 27643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:22.358613 27643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:22.358702 27643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:54:22.358771 27643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:54:22.367339 27643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:40147,127.13.148.189:43297,127.13.148.188:40779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:40147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:22.368466 27643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:22.369613 27643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:22.380851 27648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:22.380900 27649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:22.380800 27651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:22.381659 27643 server_base.cc:1034] running on GCE node
I20260806 04:54:22.382321 27643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:22.383442 27643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:22.384653 27643 hybrid_clock.cc:648] HybridClock initialized: now 1785992062384620 us; error 27 us; skew 500 ppm
I20260806 04:54:22.385032 27643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:22 dist-test-slave-1x4g krb5kdc[27629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:22.387760 27643 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:54:22.389151 27643 webserver.cc:460] Webserver started at http://127.13.148.190:44929/ using document root <none> and password file <none>
I20260806 04:54:22.389809 27643 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:22.389986 27643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:22.390261 27643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:54:22.393199 27643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "ffd9bcab984449afbdb4859bdb257521"
format_stamp: "Formatted at 2026-08-06 04:54:22 on dist-test-slave-1x4g"
server_key: "bd5809887fa033cc51a0b2912e21bafb"
server_key_iv: "914d6350a9e96395baf6813245aad838"
server_key_version: "encryptionkey@0"
I20260806 04:54:22.393905 27643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "ffd9bcab984449afbdb4859bdb257521"
format_stamp: "Formatted at 2026-08-06 04:54:22 on dist-test-slave-1x4g"
server_key: "bd5809887fa033cc51a0b2912e21bafb"
server_key_iv: "914d6350a9e96395baf6813245aad838"
server_key_version: "encryptionkey@0"
I20260806 04:54:22.398391 27643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:54:22.401528 27658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:22.402669 27643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:54:22.402839 27643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "ffd9bcab984449afbdb4859bdb257521"
format_stamp: "Formatted at 2026-08-06 04:54:22 on dist-test-slave-1x4g"
server_key: "bd5809887fa033cc51a0b2912e21bafb"
server_key_iv: "914d6350a9e96395baf6813245aad838"
server_key_version: "encryptionkey@0"
I20260806 04:54:22.403026 27643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:22.420495 27643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:22.424118 27643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:22.424364 27643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:22.441990 27643 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:40147
I20260806 04:54:22.442056 27710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:40147 every 8 connection(s)
I20260806 04:54:22.443157 27643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:54:22.446426 27711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:22.450320 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 27643
I20260806 04:54:22.450563 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:54:22.451202 13906 external_mini_cluster.cc:1428] Setting key 977223a2558a19e67b8a98bb040b90d1
I20260806 04:54:22.452540 27711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:54:22 dist-test-slave-1x4g krb5kdc[27629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.189@KRBTEST.COM" created.
I20260806 04:54:22.479368 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.456059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.13.148.190:40147)
0806 04:54:22.461831 (+ 5772us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:22.461840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:22.461888 (+ 48us) server_negotiation.cc:407] Connection header received
0806 04:54:22.463156 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:22.463165 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:22.463364 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:22.463514 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:22.464946 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:22.466039 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:22.467960 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:22.468418 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:22.468532 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:22.472493 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:22.472501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:22.472509 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:22.472540 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:22.475138 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:22.475798 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:22.475803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:22.475806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:22.475946 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:22.476326 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:22.476332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:22.476345 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:22.476891 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:22.477106 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:22.477675 (+ 569us) server_negotiation.cc:299] Negotiation successful
0806 04:54:22.477976 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5557,"spinlock_wait_cycles":54016,"thread_start_us":5331,"threads_started":1}
I20260806 04:54:22.479369 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.456062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:40147 (local address 127.0.0.1:33708)
0806 04:54:22.460741 (+ 4679us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:22.460819 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:22.462848 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:22.463588 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:22.463596 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:22.464299 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:22.464737 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:22.464761 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:22.466183 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:22.466190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:22.467835 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:22.467851 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:22.468652 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:22.468660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:22.468886 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:22.469541 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:22.469570 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:22.472263 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:22.475266 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:22.475279 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:22.475290 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:22.475662 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:22.476054 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:22.476061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:22.476066 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:22.476219 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:22.477058 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:22.477066 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:22.477279 (+ 213us) client_negotiation.cc:769] Sending connection context
0806 04:54:22.477528 (+ 249us) client_negotiation.cc:240] Negotiation successful
0806 04:54:22.478105 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4323,"mutex_wait_us":150,"thread_start_us":4069,"threads_started":1}
I20260806 04:54:22.482322 27711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:54:22.483340 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.482919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:43297 (local address 127.0.0.1:35014)
0806 04:54:22.483093 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:22.483209 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 04:54:22.483575 27661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.189:43297: connect: Connection refused (error 111)
I20260806 04:54:22.485158 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.484753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:43297 (local address 127.0.0.1:35016)
0806 04:54:22.484930 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:22.485009 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260806 04:54:22.485360 27711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.189:43297: Network error: Client connection negotiation failed: client connection to 127.13.148.189:43297: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.189.keytab.
Entry for principal kudu/127.13.148.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.189.keytab.
I20260806 04:54:22.504594 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.189:43297
--webserver_interface=127.13.148.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.189.keytab
--principal=kudu/127.13.148.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:40147,127.13.148.189:43297,127.13.148.188:40779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:54:22.534770 27711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } attempt: 1
I20260806 04:54:22.535876 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.535453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:43297 (local address 127.0.0.1:35018)
0806 04:54:22.535660 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:22.535764 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260806 04:54:22.536875 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.536471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:43297 (local address 127.0.0.1:35030)
0806 04:54:22.536622 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:22.536704 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 04:54:22.537055 27711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.189:43297: Network error: Client connection negotiation failed: client connection to 127.13.148.189:43297: connect: Connection refused (error 111)
I20260806 04:54:22.605374 27711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } attempt: 2
I20260806 04:54:22.606662 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.606076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:43297 (local address 127.0.0.1:35046)
0806 04:54:22.606364 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:22.606483 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 04:54:22.607630 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.607295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:43297 (local address 127.0.0.1:35050)
0806 04:54:22.607427 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:22.607514 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 04:54:22.607795 27711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.189:43297: Network error: Client connection negotiation failed: client connection to 127.13.148.189:43297: connect: Connection refused (error 111)
I20260806 04:54:22.699177 27711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } attempt: 3
I20260806 04:54:22.700232 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.699815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:43297 (local address 127.0.0.1:35058)
0806 04:54:22.700001 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:22.700112 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 04:54:22.701388 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.701020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:43297 (local address 127.0.0.1:35070)
0806 04:54:22.701193 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:22.701258 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 04:54:22.701599 27711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.189:43297: Network error: Client connection negotiation failed: client connection to 127.13.148.189:43297: connect: Connection refused (error 111)
W20260806 04:54:22.801570 27718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:22.801879 27718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:22.802026 27718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:22.808540 27718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:54:22.808652 27718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:22.808740 27718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:22.808841 27718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:54:22.808900 27718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:54:22.817616 27718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:40147,127.13.148.189:43297,127.13.148.188:40779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.189.keytab
--principal=kudu/127.13.148.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.189:43297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.148.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:22.818815 27718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:22.819989 27718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:22.832554 27724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:22.833026 27723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:22.833237 27726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:22.834161 27718 server_base.cc:1034] running on GCE node
I20260806 04:54:22.834894 27718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:22.836130 27718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:22.837348 27718 hybrid_clock.cc:648] HybridClock initialized: now 1785992062837313 us; error 48 us; skew 500 ppm
I20260806 04:54:22.837692 27718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:22 dist-test-slave-1x4g krb5kdc[27629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:22.840708 27718 init.cc:377] Logged in from keytab as kudu/127.13.148.189@KRBTEST.COM (short username kudu)
I20260806 04:54:22.842346 27718 webserver.cc:460] Webserver started at http://127.13.148.189:33511/ using document root <none> and password file <none>
I20260806 04:54:22.843261 27718 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:22.843417 27718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:22.843761 27718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:54:22.844868 27711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } attempt: 4
I20260806 04:54:22.845846 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.845393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:43297 (local address 127.0.0.1:35072)
0806 04:54:22.845645 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:22.845726 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260806 04:54:22.846776 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.846436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:43297 (local address 127.0.0.1:35074)
0806 04:54:22.846579 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:22.846668 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 04:54:22.846701 27718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-1/data/instance:
uuid: "dcafff36483144a2ac19268626d9cd09"
format_stamp: "Formatted at 2026-08-06 04:54:22 on dist-test-slave-1x4g"
server_key: "0a273d6c8ebb7c752636b90321ff4b4d"
server_key_iv: "7976c19434a18cda4a9bcabdef5858c5"
server_key_version: "encryptionkey@0"
W20260806 04:54:22.846978 27711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.189:43297: Network error: Client connection negotiation failed: client connection to 127.13.148.189:43297: connect: Connection refused (error 111)
I20260806 04:54:22.847456 27718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-1/wal/instance:
uuid: "dcafff36483144a2ac19268626d9cd09"
format_stamp: "Formatted at 2026-08-06 04:54:22 on dist-test-slave-1x4g"
server_key: "0a273d6c8ebb7c752636b90321ff4b4d"
server_key_iv: "7976c19434a18cda4a9bcabdef5858c5"
server_key_version: "encryptionkey@0"
I20260806 04:54:22.852020 27718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:54:22.855027 27733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:22.856221 27718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 04:54:22.856379 27718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-1/wal
uuid: "dcafff36483144a2ac19268626d9cd09"
format_stamp: "Formatted at 2026-08-06 04:54:22 on dist-test-slave-1x4g"
server_key: "0a273d6c8ebb7c752636b90321ff4b4d"
server_key_iv: "7976c19434a18cda4a9bcabdef5858c5"
server_key_version: "encryptionkey@0"
I20260806 04:54:22.856571 27718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:22.866990 27718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:22.870517 27718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:22.870775 27718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:22.887775 27718 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.189:43297
I20260806 04:54:22.887817 27785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.189:43297 every 8 connection(s)
I20260806 04:54:22.889101 27718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-1/data/info.pb
I20260806 04:54:22.892403 27786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:22.894176 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 27718
I20260806 04:54:22.894356 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-1/wal/instance
I20260806 04:54:22.894791 13906 external_mini_cluster.cc:1428] Setting key 200d1746a491565f0c1c93290bd56167
I20260806 04:54:22.898198 27786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:54:22 dist-test-slave-1x4g krb5kdc[27629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.189@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.188@KRBTEST.COM; defaulting to no policy
I20260806 04:54:22.918737 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.900568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.13.148.190:40147)
0806 04:54:22.900824 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:22.900831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:22.902025 (+ 1194us) server_negotiation.cc:407] Connection header received
0806 04:54:22.903764 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:22.903772 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:22.903855 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:22.903970 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:22.905459 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:22.906461 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:22.907923 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:22.908587 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:22.908719 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:22.912109 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:22.912116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:22.912123 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:22.912164 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:22.915589 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:22.916397 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:22.916403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:22.916409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:22.916565 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:22.917085 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:22.917091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:22.917094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:22.917519 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:22.917697 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:22.918336 (+ 639us) server_negotiation.cc:299] Negotiation successful
0806 04:54:22.918529 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 04:54:22.919533 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.901038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:40147 (local address 127.0.0.1:33720)
0806 04:54:22.901844 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:22.901886 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:22.903507 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:22.904196 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:22.904202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:22.904635 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:22.905276 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:22.905293 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:22.906616 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:22.906619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:22.907797 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:22.907809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:22.908765 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:22.908769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:22.909013 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:22.909631 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:22.909650 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:22.911888 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:22.915749 (+ 3861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:22.915755 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:22.915765 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:22.916266 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:22.916699 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:22.916702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:22.916704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:22.916923 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:22.917651 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:22.917657 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:22.917969 (+ 312us) client_negotiation.cc:769] Sending connection context
0806 04:54:22.918192 (+ 223us) client_negotiation.cc:240] Negotiation successful
0806 04:54:22.918487 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":349,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.188@KRBTEST.COM" created.
I20260806 04:54:22.921276 27786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:54:22 dist-test-slave-1x4g krb5kdc[27629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.189@KRBTEST.COM for kudu/127.13.148.189@KRBTEST.COM
I20260806 04:54:22.933997 27790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.922160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.13.148.189:43297)
0806 04:54:22.922738 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:22.922743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:22.922765 (+ 22us) server_negotiation.cc:407] Connection header received
0806 04:54:22.922804 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:22.922809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:22.922941 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:22.923092 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:22.923689 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:22.924537 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:22.925579 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:22.926036 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:22.926157 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:22.928919 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:22.928929 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:22.928939 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:22.928965 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:22.931365 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:22.932132 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:22.932137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:22.932139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:22.932232 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:22.932591 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:22.932594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:22.932596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:22.932939 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:22.933059 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:22.933486 (+ 427us) server_negotiation.cc:299] Negotiation successful
0806 04:54:22.933744 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":456,"thread_start_us":342,"threads_started":1}
I20260806 04:54:22.934084 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.921840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:43297 (local address 127.0.0.1:35078)
0806 04:54:22.922017 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:22.922031 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:22.922116 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:22.923082 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:22.923086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:22.923449 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:22.923553 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:22.923560 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:22.924659 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:22.924663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:22.925434 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:22.925444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:22.926162 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:22.926167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:22.926404 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:22.926903 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:22.926917 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:22.928705 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:22.931523 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:22.931531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:22.931537 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:22.932023 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:22.932325 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:22.932332 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:22.932337 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:22.932500 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:22.933050 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:22.933057 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:22.933223 (+ 166us) client_negotiation.cc:769] Sending connection context
0806 04:54:22.933369 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 04:54:22.933647 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":20}
I20260806 04:54:22.936146 27786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:54:22.937211 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.936685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38224)
0806 04:54:22.936895 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:22.937031 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 04:54:22.937461 27737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
I20260806 04:54:22.938963 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.938546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38238)
0806 04:54:22.938727 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:22.938824 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 04:54:22.939185 27786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:40779: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.188.keytab.
Entry for principal kudu/127.13.148.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.188.keytab.
I20260806 04:54:22.949954 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.188:40779
--webserver_interface=127.13.148.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.188.keytab
--principal=kudu/127.13.148.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:40147,127.13.148.189:43297,127.13.148.188:40779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:54:22.988544 27786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } attempt: 1
I20260806 04:54:22.989686 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.989230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38242)
0806 04:54:22.989392 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:22.989503 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 04:54:22.990609 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:22.990258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38246)
0806 04:54:22.990449 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:22.990511 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 04:54:22.990859 27786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:40779: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
I20260806 04:54:23.059147 27786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } attempt: 2
I20260806 04:54:23.060292 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.059783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38252)
0806 04:54:23.059992 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.060102 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 04:54:23.061431 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.061030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38268)
0806 04:54:23.061185 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.061277 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 04:54:23.061695 27786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:40779: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
I20260806 04:54:23.146224 27711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } attempt: 5
I20260806 04:54:23.152902 27786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } attempt: 3
I20260806 04:54:23.153878 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.153366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38284)
0806 04:54:23.153577 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.153689 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 04:54:23.154966 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.154480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38288)
0806 04:54:23.154707 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.154802 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260806 04:54:23.155194 27786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:40779: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Aug 06 04:54:23 dist-test-slave-1x4g krb5kdc[27629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for kudu/127.13.148.189@KRBTEST.COM
I20260806 04:54:23.164423 27790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.147051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35088 (local address 127.13.148.189:43297)
0806 04:54:23.147254 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:23.147259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:23.147282 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:54:23.147345 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:23.147350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:23.147420 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:23.147534 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:23.148282 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.149288 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.150291 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.150982 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.151141 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.159356 (+ 8215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:23.159365 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:23.159371 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:23.159427 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:23.162164 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.162881 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.162884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.162886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.162985 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.163370 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.163372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.163374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.163603 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:23.163698 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:23.164132 (+ 434us) server_negotiation.cc:299] Negotiation successful
0806 04:54:23.164267 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 04:54:23.164671 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.146954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:43297 (local address 127.0.0.1:35088)
0806 04:54:23.147136 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.147156 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:23.147264 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:23.147568 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:23.147572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:23.147911 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:23.148094 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:23.148103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.149406 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.149410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:23.150151 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:23.150162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.151121 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.151126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:23.151268 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.156710 (+ 5442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:23.156728 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:23.159119 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:23.162294 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:23.162301 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:23.162307 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:23.162785 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:23.163077 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:23.163090 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:23.163095 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:23.163263 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:23.163711 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:23.163717 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:23.163903 (+ 186us) client_negotiation.cc:769] Sending connection context
0806 04:54:23.164053 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 04:54:23.164331 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 04:54:23.165632 27711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:54:23.166710 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.166297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38290)
0806 04:54:23.166472 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.166564 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 04:54:23.167642 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.167265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38298)
0806 04:54:23.167415 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.167517 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 04:54:23.167840 27711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:40779: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
I20260806 04:54:23.219075 27711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } attempt: 1
I20260806 04:54:23.220299 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.219736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38306)
0806 04:54:23.219961 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.220097 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 04:54:23.221383 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.220958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38312)
0806 04:54:23.221146 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.221250 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 04:54:23.221626 27711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:40779: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
W20260806 04:54:23.254657 27793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:23.254886 27793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:23.254963 27793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:23.261440 27793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:54:23.261569 27793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:23.261703 27793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:23.261795 27793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:54:23.261866 27793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:54:23.269917 27793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:40147,127.13.148.189:43297,127.13.148.188:40779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.188.keytab
--principal=kudu/127.13.148.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.188:40779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.148.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:23.271087 27793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:23.272259 27793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:23.283135 27799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:23.283136 27802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:23.283167 27800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:23.284109 27793 server_base.cc:1034] running on GCE node
I20260806 04:54:23.284919 27793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:23.286079 27793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:23.287269 27793 hybrid_clock.cc:648] HybridClock initialized: now 1785992063287230 us; error 32 us; skew 500 ppm
I20260806 04:54:23.287564 27793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:54:23.289824 27711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } attempt: 2
Aug 06 04:54:23 dist-test-slave-1x4g krb5kdc[27629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992063, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:23.290810 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.290370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38316)
0806 04:54:23.290574 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.290664 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 04:54:23.291337 27793 init.cc:377] Logged in from keytab as kudu/127.13.148.188@KRBTEST.COM (short username kudu)
I20260806 04:54:23.291973 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.291445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38328)
0806 04:54:23.291721 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.291817 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260806 04:54:23.292229 27711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:40779: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
I20260806 04:54:23.293135 27793 webserver.cc:460] Webserver started at http://127.13.148.188:42845/ using document root <none> and password file <none>
I20260806 04:54:23.293673 27793 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:23.293762 27793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:23.293996 27793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:54:23.296486 27793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-2/data/instance:
uuid: "c80ef8e9a7534a489c2211e6f2c938b1"
format_stamp: "Formatted at 2026-08-06 04:54:23 on dist-test-slave-1x4g"
server_key: "09fae8e6d28e18a2444f9a4d0e893a35"
server_key_iv: "67f3ce62f2fb15ea9eeef1f493ffb6a7"
server_key_version: "encryptionkey@0"
I20260806 04:54:23.297328 27793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-2/wal/instance:
uuid: "c80ef8e9a7534a489c2211e6f2c938b1"
format_stamp: "Formatted at 2026-08-06 04:54:23 on dist-test-slave-1x4g"
server_key: "09fae8e6d28e18a2444f9a4d0e893a35"
server_key_iv: "67f3ce62f2fb15ea9eeef1f493ffb6a7"
server_key_version: "encryptionkey@0"
I20260806 04:54:23.298380 27786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } attempt: 4
I20260806 04:54:23.299369 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.298835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38338)
0806 04:54:23.299091 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.299197 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 04:54:23.300302 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.299914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38342)
0806 04:54:23.300067 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.300151 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 04:54:23.300586 27786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:40779: Network error: Client connection negotiation failed: client connection to 127.13.148.188:40779: connect: Connection refused (error 111)
I20260806 04:54:23.302480 27793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260806 04:54:23.305510 27809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:23.306699 27793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:54:23.306861 27793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-2/wal
uuid: "c80ef8e9a7534a489c2211e6f2c938b1"
format_stamp: "Formatted at 2026-08-06 04:54:23 on dist-test-slave-1x4g"
server_key: "09fae8e6d28e18a2444f9a4d0e893a35"
server_key_iv: "67f3ce62f2fb15ea9eeef1f493ffb6a7"
server_key_version: "encryptionkey@0"
I20260806 04:54:23.307032 27793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:23.332592 27793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:23.336138 27793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:23.336417 27793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:23.353277 27793 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.188:40779
I20260806 04:54:23.353294 27862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.188:40779 every 8 connection(s)
I20260806 04:54:23.354393 27793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-2/data/info.pb
I20260806 04:54:23.357559 27863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:23.359071 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 27793
I20260806 04:54:23.359318 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991872418975-13906-0/minicluster-data/master-2/wal/instance
I20260806 04:54:23.359746 13906 external_mini_cluster.cc:1428] Setting key 23d0c2ccf8a432886e65b06724a3101f
I20260806 04:54:23.366470 27863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:54:23 dist-test-slave-1x4g krb5kdc[27629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992063, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.188@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
Aug 06 04:54:23 dist-test-slave-1x4g krb5kdc[27629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.188@KRBTEST.COM
I20260806 04:54:23.384310 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.368967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.13.148.190:40147)
0806 04:54:23.369137 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:23.369141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:23.369806 (+ 665us) server_negotiation.cc:407] Connection header received
0806 04:54:23.369957 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:23.369961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:23.370020 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:23.370089 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:23.371254 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.371920 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.373117 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.373493 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.373588 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.378466 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:23.378470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:23.378473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:23.378502 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:23.381196 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.382063 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.382066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.382068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.382181 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.382647 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.382652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.382654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.382962 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:23.383111 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:23.383881 (+ 770us) server_negotiation.cc:299] Negotiation successful
0806 04:54:23.384024 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:54:23.385332 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.368967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:40147 (local address 127.0.0.1:33728)
0806 04:54:23.369656 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.369691 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:23.369838 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:23.370176 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:23.370182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:23.370863 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:23.371119 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:23.371128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.372040 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.372043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:23.373000 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:23.373012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.373667 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.373671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:23.373767 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.376125 (+ 2358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:23.376143 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:23.378298 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:23.381318 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:23.381330 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:23.381343 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:23.381956 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:23.382300 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:23.382305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:23.382308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:23.382524 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:23.383075 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:23.383080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:23.383550 (+ 470us) client_negotiation.cc:769] Sending connection context
0806 04:54:23.383783 (+ 233us) client_negotiation.cc:240] Negotiation successful
0806 04:54:23.384191 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"spinlock_wait_cycles":14080,"thread_start_us":368,"threads_started":1}
I20260806 04:54:23.385600 27866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.361475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.13.148.188:40779)
0806 04:54:23.362448 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:23.362468 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:23.362520 (+ 52us) server_negotiation.cc:407] Connection header received
0806 04:54:23.364014 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:23.364041 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:23.364471 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:23.364875 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:23.366061 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.367831 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.369292 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.370230 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.370413 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.379429 (+ 9016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:23.379436 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:23.379447 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:23.379471 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:23.382410 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.383218 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.383224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.383227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.383376 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.383755 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.383759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.383762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.384145 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:23.384411 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:23.384980 (+ 569us) server_negotiation.cc:299] Negotiation successful
0806 04:54:23.385182 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":450,"threads_started":1}
I20260806 04:54:23.386811 27863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:54:23 dist-test-slave-1x4g krb5kdc[27629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992063, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.188@KRBTEST.COM for kudu/127.13.148.189@KRBTEST.COM
Aug 06 04:54:23 dist-test-slave-1x4g krb5kdc[27629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:23.398455 27711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } attempt: 3
I20260806 04:54:23.400090 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.387388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:43297 (local address 127.0.0.1:35098)
0806 04:54:23.387550 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.387581 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:23.387668 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:23.388115 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:23.388121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:23.388504 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:23.388659 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:23.388669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.389993 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.389996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:23.391380 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:23.391391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.392265 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.392269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:23.392398 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.392958 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:23.392972 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:23.394793 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:23.397773 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:23.397779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:23.397783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:23.398122 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:23.398530 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:23.398536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:23.398539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:23.398711 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:23.399231 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:23.399235 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:23.399384 (+ 149us) client_negotiation.cc:769] Sending connection context
0806 04:54:23.399534 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 04:54:23.399777 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 04:54:23.400288 27790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.387620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.13.148.189:43297)
0806 04:54:23.387813 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:23.387817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:23.387831 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:54:23.387877 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:23.387880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:23.387963 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:23.388061 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:23.388828 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.389873 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.391506 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.392145 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.392257 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.394976 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:23.394980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:23.394982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:23.395009 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:23.397636 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.398241 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.398247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.398250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.398410 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.398826 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.398830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.398832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.399113 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:23.399251 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:23.399634 (+ 383us) server_negotiation.cc:299] Negotiation successful
0806 04:54:23.399834 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 04:54:23.401286 27863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:54:23.403390 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.388253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.13.148.190:40147)
0806 04:54:23.388406 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:23.388413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:23.388428 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:54:23.388506 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:23.388510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:23.388569 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:23.388663 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:23.390092 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.391200 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.392848 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.393221 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.393340 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.396859 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:23.396864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:23.396867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:23.396895 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:23.399793 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.400575 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.400581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.400584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.400753 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.401295 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.401298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.401300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.401553 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:23.401850 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:23.402692 (+ 842us) server_negotiation.cc:299] Negotiation successful
0806 04:54:23.402946 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 04:54:23 dist-test-slave-1x4g krb5kdc[27629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for kudu/127.13.148.188@KRBTEST.COM
Aug 06 04:54:23 dist-test-slave-1x4g krb5kdc[27629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.189@KRBTEST.COM
I20260806 04:54:23.415869 27866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.398976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.13.148.188:40779)
0806 04:54:23.399451 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:23.399454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:23.399465 (+ 11us) server_negotiation.cc:407] Connection header received
0806 04:54:23.399494 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:23.399516 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:23.399573 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:23.400767 (+ 1194us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:23.400972 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.402558 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.403796 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.404309 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.404421 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.408090 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:23.408095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:23.408099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:23.408129 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:23.412637 (+ 4508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.414066 (+ 1429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.414073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.414079 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.414236 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.414573 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.414578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.414582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.414873 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:23.415026 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:23.415388 (+ 362us) server_negotiation.cc:299] Negotiation successful
0806 04:54:23.415564 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398}
I20260806 04:54:23.416126 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.398962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38360)
0806 04:54:23.399219 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.399236 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:23.399313 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:23.399711 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:23.399715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:23.400098 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:23.400300 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:23.400314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.402717 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.402722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:23.403669 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:23.403682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.404434 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.404448 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:23.404600 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.405259 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:23.405280 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:23.407900 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:23.412803 (+ 4903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:23.412808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:23.412811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:23.413241 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:23.414347 (+ 1106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:23.414350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:23.414352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:23.414470 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:23.414999 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:23.415003 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:23.415170 (+ 167us) client_negotiation.cc:769] Sending connection context
0806 04:54:23.415304 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 04:54:23.415562 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
Aug 06 04:54:23 dist-test-slave-1x4g krb5kdc[27629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992063, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.188@KRBTEST.COM for kudu/127.13.148.188@KRBTEST.COM
I20260806 04:54:23.422466 27790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.405592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35114 (local address 127.13.148.189:43297)
0806 04:54:23.406409 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:23.406412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:23.406424 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:54:23.406495 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:23.406499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:23.406549 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:23.406636 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:23.407942 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.408899 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.409964 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.410601 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.410738 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.415813 (+ 5075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:23.415823 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:23.415829 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:23.415876 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:23.419459 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.420307 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.420312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.420315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.420493 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.420978 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.420984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.420987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.421338 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:23.421627 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:23.421904 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 04:54:23.422122 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731}
I20260806 04:54:23.422644 27711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521: Bootstrap starting.
I20260806 04:54:23.426709 27711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:23.427098 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:54:23.427829 27711 log.cc:824] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:23.428947 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.402277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38376)
0806 04:54:23.402474 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.402491 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:23.402564 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:23.416497 (+ 13933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:23.416501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:23.417435 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:23.417604 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:23.417616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.418580 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.418586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:23.419328 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:23.419338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.420028 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.420034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:23.420213 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.420723 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:23.420738 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:23.423049 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:23.426297 (+ 3248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:23.426302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:23.426305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:23.426847 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:23.427229 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:23.427234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:23.427237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:23.427378 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:23.427915 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:23.427919 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:23.428069 (+ 150us) client_negotiation.cc:769] Sending connection context
0806 04:54:23.428242 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 04:54:23.428611 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260806 04:54:23.429230 27866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.402299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38376 (local address 127.13.148.188:40779)
0806 04:54:23.416153 (+ 13854us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:23.416158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:23.416174 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:54:23.416244 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:23.416250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:23.416305 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:23.416382 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:23.417744 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.418463 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.419487 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.419891 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.420018 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.423212 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:23.423217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:23.423221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:23.423249 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:23.426189 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.426957 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.426960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.426964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.427112 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.427490 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.427494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.427496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.427786 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:23.427911 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:23.428347 (+ 436us) server_negotiation.cc:299] Negotiation successful
0806 04:54:23.428748 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":194,"server-negotiator.queue_time_us":13759,"thread_start_us":15114,"threads_started":1}
I20260806 04:54:23.431247 27711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521: No bootstrap required, opened a new log
I20260806 04:54:23.434114 27863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1: Bootstrap starting.
I20260806 04:54:23.438117 27863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:23.439605 27863 log.cc:824] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:23.441639 27711 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } }
I20260806 04:54:23.441998 27711 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:23.442147 27711 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd9bcab984449afbdb4859bdb257521, State: Initialized, Role: FOLLOWER
I20260806 04:54:23.442790 27711 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } }
I20260806 04:54:23.448660 27863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1: No bootstrap required, opened a new log
I20260806 04:54:23.449568 27879 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } } }
I20260806 04:54:23.449909 27879 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:54:23.451390 27711 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:54:23.458351 27866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.441321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38392 (local address 127.13.148.188:40779)
0806 04:54:23.441466 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:23.441471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:23.442725 (+ 1254us) server_negotiation.cc:407] Connection header received
0806 04:54:23.442961 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:23.442968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:23.443060 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:23.443127 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:23.445249 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.446327 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.447967 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.448548 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.448627 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.452024 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:23.452030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:23.452035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:23.452068 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:23.455272 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.456013 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.456021 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.456024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.456152 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.456545 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.456549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.456552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.456954 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:23.457184 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:23.457813 (+ 629us) server_negotiation.cc:299] Negotiation successful
0806 04:54:23.458021 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:54:23.463871 27863 raft_consensus.cc:348] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } }
I20260806 04:54:23.464291 27863 raft_consensus.cc:374] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:23.464455 27863 raft_consensus.cc:729] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c80ef8e9a7534a489c2211e6f2c938b1, State: Initialized, Role: FOLLOWER
I20260806 04:54:23.465109 27863 consensus_queue.cc:260] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } }
I20260806 04:54:23.471295 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.434871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.13.148.190:40147)
0806 04:54:23.435014 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:23.435019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:23.435183 (+ 164us) server_negotiation.cc:407] Connection header received
0806 04:54:23.435416 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:23.435420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:23.435479 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:23.435555 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:23.437795 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.438956 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.440927 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.441558 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.441695 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.462449 (+ 20754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:23.462459 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:23.462467 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:23.462545 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:23.466734 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.468384 (+ 1650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.468388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.468391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.468549 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.469060 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.469067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.469072 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.469467 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:23.469751 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:23.470410 (+ 659us) server_negotiation.cc:299] Negotiation successful
0806 04:54:23.470753 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:54:23.473491 27863 sys_catalog.cc:564] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:54:23.474815 27790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.435696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35126 (local address 127.13.148.189:43297)
0806 04:54:23.439869 (+ 4173us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:23.439874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:23.442112 (+ 2238us) server_negotiation.cc:407] Connection header received
0806 04:54:23.443091 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:23.443098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:23.443163 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:23.443221 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:23.456952 (+ 13731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.458031 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.462388 (+ 4357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.462969 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.463078 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.466666 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:23.466674 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:23.466681 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:23.466729 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:23.471953 (+ 5224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.472740 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.472744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.472748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.472921 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.473339 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.473343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.473345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.473732 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:23.473965 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:23.474287 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 04:54:23.474497 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4072}
I20260806 04:54:23.473381 27884 sys_catalog.cc:455] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } } }
I20260806 04:54:23.475193 27884 sys_catalog.cc:458] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:54:23.476354 27896 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260806 04:54:23.487331 27902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:54:23.487517 27902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:54:23.481457 27643 master_runner.cc:186] Error getting master registration for 127.13.148.189:43297: OK, instance_id { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" instance_seqno: 1785992062882955 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 04:54:23.492506 27643 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:54:23.485188 27793 master_runner.cc:186] Error getting master registration for 127.13.148.189:43297: OK, instance_id { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" instance_seqno: 1785992062882955 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 04:54:23.493731 27793 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:54:23.500988 27879 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:54:23.501238 27879 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } }
I20260806 04:54:23.502795 27879 leader_election.cc:288] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcafff36483144a2ac19268626d9cd09 (127.13.148.189:43297), c80ef8e9a7534a489c2211e6f2c938b1 (127.13.148.188:40779)
I20260806 04:54:23.503767 27838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffd9bcab984449afbdb4859bdb257521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" is_pre_election: true
I20260806 04:54:23.504295 27838 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffd9bcab984449afbdb4859bdb257521 in term 0.
I20260806 04:54:23.504849 27662 leader_election.cc:302] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c80ef8e9a7534a489c2211e6f2c938b1, ffd9bcab984449afbdb4859bdb257521; no voters:
I20260806 04:54:23.505172 27879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:54:23.505308 27879 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:54:23.505400 27879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:23.508154 27879 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } }
I20260806 04:54:23.508731 27879 leader_election.cc:288] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [CANDIDATE]: Term 1 election: Requested vote from peers dcafff36483144a2ac19268626d9cd09 (127.13.148.189:43297), c80ef8e9a7534a489c2211e6f2c938b1 (127.13.148.188:40779)
I20260806 04:54:23.509443 27838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffd9bcab984449afbdb4859bdb257521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c80ef8e9a7534a489c2211e6f2c938b1"
I20260806 04:54:23.509665 27838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:23.513010 27838 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffd9bcab984449afbdb4859bdb257521 in term 1.
I20260806 04:54:23.513455 27662 leader_election.cc:302] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c80ef8e9a7534a489c2211e6f2c938b1, ffd9bcab984449afbdb4859bdb257521; no voters:
I20260806 04:54:23.513912 27879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:54:23.514750 27879 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [term 1 LEADER]: Becoming Leader. State: Replica: ffd9bcab984449afbdb4859bdb257521, State: Running, Role: LEADER
I20260806 04:54:23.515255 27879 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } }
I20260806 04:54:23.518275 27905 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd9bcab984449afbdb4859bdb257521. Latest consensus state: current_term: 1 leader_uuid: "ffd9bcab984449afbdb4859bdb257521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } } }
I20260806 04:54:23.518639 27905 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [sys.catalog]: This master's current role is: LEADER
I20260806 04:54:23.519549 27907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:54:23.524392 27907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:54:23.533627 27838 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1 [term 1 FOLLOWER]: Refusing update from remote peer ffd9bcab984449afbdb4859bdb257521: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:54:23.534375 27905 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:54:23.560113 27884 sys_catalog.cc:455] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd9bcab984449afbdb4859bdb257521. Latest consensus state: current_term: 1 leader_uuid: "ffd9bcab984449afbdb4859bdb257521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } } }
I20260806 04:54:23.560432 27884 sys_catalog.cc:458] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:54:23.563414 27909 mvcc.cc:205] Tried to move back new op lower bound from 7315423492225617920 to 7315423492161953792. Current Snapshot: MvccSnapshot[applied={T|T < 7315423492225617920}]
I20260806 04:54:23.569211 27905 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ffd9bcab984449afbdb4859bdb257521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } } }
I20260806 04:54:23.569453 27905 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [sys.catalog]: This master's current role is: LEADER
I20260806 04:54:23.571432 27907 catalog_manager.cc:1347] Generated new cluster ID: f2b103e60b094368914d0b6d6d917c95
I20260806 04:54:23.571554 27907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:54:23.578702 27884 sys_catalog.cc:455] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ffd9bcab984449afbdb4859bdb257521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } } }
I20260806 04:54:23.578888 27884 sys_catalog.cc:458] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:54:23.590317 27907 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:54:23.599241 27907 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:54:23.599794 27786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } attempt: 5
Aug 06 04:54:23 dist-test-slave-1x4g krb5kdc[27629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.189@KRBTEST.COM for kudu/127.13.148.188@KRBTEST.COM
I20260806 04:54:23.625548 27866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.601042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38396 (local address 127.13.148.188:40779)
0806 04:54:23.601192 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:23.601200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:23.601218 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:54:23.601273 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:23.601278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:23.601354 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:23.601455 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:23.602511 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.603469 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.604978 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.605533 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.605622 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.610152 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:23.610168 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:23.610172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:23.610204 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:23.618673 (+ 8469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.620146 (+ 1473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.620155 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.620158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.620465 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:23.621457 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:23.621461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:23.621464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:23.622336 (+ 872us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:23.622839 (+ 503us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:23.623665 (+ 826us) server_negotiation.cc:299] Negotiation successful
0806 04:54:23.624255 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 04:54:23.626741 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:23.600306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:40779 (local address 127.0.0.1:38396)
0806 04:54:23.600657 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:23.600677 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:23.600780 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:23.601671 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:23.601676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:23.602169 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:23.602361 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:23.602374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.603622 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.603627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:23.604835 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:23.604848 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:23.605835 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:23.605840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:23.606014 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:23.606759 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:23.606778 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:23.609845 (+ 3067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:23.618916 (+ 9071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:23.618942 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:23.618945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:23.619932 (+ 987us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:23.620811 (+ 879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:23.620818 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:23.620823 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:23.621229 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:23.622961 (+ 1732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:23.622967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:23.623206 (+ 239us) client_negotiation.cc:769] Sending connection context
0806 04:54:23.624637 (+ 1431us) client_negotiation.cc:240] Negotiation successful
0806 04:54:23.625959 (+ 1322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232}
I20260806 04:54:23.634652 27786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09: Bootstrap starting.
I20260806 04:54:23.642263 27786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:23.648114 27907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521: Generated new TSK 0
I20260806 04:54:23.649363 27786 log.cc:824] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:23.650079 27907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:54:23.658545 27786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09: No bootstrap required, opened a new log
I20260806 04:54:23.682554 27786 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } }
I20260806 04:54:23.683127 27786 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:23.683301 27786 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcafff36483144a2ac19268626d9cd09, State: Initialized, Role: FOLLOWER
I20260806 04:54:23.684130 27786 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } }
I20260806 04:54:23.701465 27917 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } } }
I20260806 04:54:23.701926 27917 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:54:23.702891 27786 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:54:23.703238 27758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:23.707618 27758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09 [term 1 FOLLOWER]: Refusing update from remote peer ffd9bcab984449afbdb4859bdb257521: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:54:23.709040 27879 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:54:23.710564 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffd9bcab984449afbdb4859bdb257521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcafff36483144a2ac19268626d9cd09"
I20260806 04:54:23.712991 27760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffd9bcab984449afbdb4859bdb257521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcafff36483144a2ac19268626d9cd09" is_pre_election: true
I20260806 04:54:23.716325 27759 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffd9bcab984449afbdb4859bdb257521 in term 1.
I20260806 04:54:23.761869 27917 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd9bcab984449afbdb4859bdb257521. Latest consensus state: current_term: 1 leader_uuid: "ffd9bcab984449afbdb4859bdb257521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } } }
I20260806 04:54:23.762174 27917 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:54:23.761911 27925 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ffd9bcab984449afbdb4859bdb257521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } } }
I20260806 04:54:23.762329 27925 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:54:23.779579 27935 catalog_manager.cc:1259] Loaded cluster ID: f2b103e60b094368914d0b6d6d917c95
I20260806 04:54:23.779711 27935 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09: loading cluster ID for follower catalog manager: success
I20260806 04:54:23.783109 27935 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09: acquiring CA information for follower catalog manager: success
I20260806 04:54:23.785283 27935 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dcafff36483144a2ac19268626d9cd09: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:54:24.235808 27879 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ffd9bcab984449afbdb4859bdb257521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd9bcab984449afbdb4859bdb257521" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 40147 } } peers { permanent_uuid: "dcafff36483144a2ac19268626d9cd09" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 43297 } } peers { permanent_uuid: "c80ef8e9a7534a489c2211e6f2c938b1" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 40779 } } }
I20260806 04:54:24.236156 27879 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd9bcab984449afbdb4859bdb257521 [sys.catalog]: This master's current role is: LEADER
I20260806 04:54:24.489526 27902 catalog_manager.cc:1259] Loaded cluster ID: f2b103e60b094368914d0b6d6d917c95
I20260806 04:54:24.489668 27902 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1: loading cluster ID for follower catalog manager: success
I20260806 04:54:24.492020 27902 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1: acquiring CA information for follower catalog manager: success
I20260806 04:54:24.493690 27902 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c80ef8e9a7534a489c2211e6f2c938b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:54:24.789417 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 27643
I20260806 04:54:24.790625 27706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:24.904493 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 27643
I20260806 04:54:24.927961 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 27718
I20260806 04:54:24.929010 27781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:25.032665 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 27718
I20260806 04:54:25.054347 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 27793
I20260806 04:54:25.055325 27858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:25.152562 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 27793
2026-08-06T04:54:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4303 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:54:25 dist-test-slave-1x4g krb5kdc[27946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:54:25 dist-test-slave-1x4g krb5kdc[27946](info): set up 2 sockets
Aug 06 04:54:25 dist-test-slave-1x4g krb5kdc[27946](info): commencing operation
krb5kdc: starting...
W20260806 04:54:26.149060 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:54:26 dist-test-slave-1x4g krb5kdc[27946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:54:26.356042 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:43547
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:33167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:43547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:54:26.656659 27959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:26.656973 27959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:26.657132 27959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:26.663508 27959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:54:26.663625 27959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:26.663715 27959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:26.663787 27959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:54:26.663853 27959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:54:26.671972 27959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:33167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:43547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:43547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:26.673043 27959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:26.674134 27959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:26.684737 27964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:26.685045 27965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:26.685246 27967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:26.686189 27959 server_base.cc:1034] running on GCE node
I20260806 04:54:26.686957 27959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:26.688100 27959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:26.689281 27959 hybrid_clock.cc:648] HybridClock initialized: now 1785992066689240 us; error 35 us; skew 500 ppm
I20260806 04:54:26.689543 27959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:26 dist-test-slave-1x4g krb5kdc[27946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992066, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:26.692305 27959 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:54:26.693569 27959 webserver.cc:460] Webserver started at http://127.13.148.190:38179/ using document root <none> and password file <none>
I20260806 04:54:26.694177 27959 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:26.694267 27959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:26.694479 27959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:54:26.697034 27959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "3c2ee6154f6844989945d52dfaee3eea"
format_stamp: "Formatted at 2026-08-06 04:54:26 on dist-test-slave-1x4g"
server_key: "46f9471d1045f16631a04b0afc76df07"
server_key_iv: "2cfbb913e806cc7b7ad85d261f040561"
server_key_version: "encryptionkey@0"
I20260806 04:54:26.697677 27959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "3c2ee6154f6844989945d52dfaee3eea"
format_stamp: "Formatted at 2026-08-06 04:54:26 on dist-test-slave-1x4g"
server_key: "46f9471d1045f16631a04b0afc76df07"
server_key_iv: "2cfbb913e806cc7b7ad85d261f040561"
server_key_version: "encryptionkey@0"
I20260806 04:54:26.702183 27959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 04:54:26.705173 27974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:26.706266 27959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260806 04:54:26.706426 27959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "3c2ee6154f6844989945d52dfaee3eea"
format_stamp: "Formatted at 2026-08-06 04:54:26 on dist-test-slave-1x4g"
server_key: "46f9471d1045f16631a04b0afc76df07"
server_key_iv: "2cfbb913e806cc7b7ad85d261f040561"
server_key_version: "encryptionkey@0"
I20260806 04:54:26.706583 27959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:26.717696 27959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:26.720988 27959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:26.721253 27959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:26.737722 27959 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:43547
I20260806 04:54:26.737751 28026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:43547 every 8 connection(s)
I20260806 04:54:26.739012 27959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:54:26.742391 28027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:26.744998 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 27959
I20260806 04:54:26.745191 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:54:26.745790 13906 external_mini_cluster.cc:1428] Setting key 6cd36d373a6fdb4c1b8a6120d65cf52d
I20260806 04:54:26.753813 28027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea: Bootstrap starting.
I20260806 04:54:26.757119 28027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:26.758085 28027 log.cc:824] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:54:26 dist-test-slave-1x4g krb5kdc[27946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:26.761178 28027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea: No bootstrap required, opened a new log
I20260806 04:54:26.767612 28030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:26.747353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.13.148.190:43547)
0806 04:54:26.748079 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:26.748092 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:26.748134 (+ 42us) server_negotiation.cc:407] Connection header received
0806 04:54:26.750014 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:26.750041 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:26.750489 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:26.750953 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:26.752252 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:26.754023 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:26.755269 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:26.756174 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:26.756348 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:26.760878 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:26.760887 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:26.760900 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:26.760935 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:26.763830 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:26.764607 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:26.764613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:26.764618 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:26.764768 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:26.765248 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:26.765250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:26.765252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:26.765720 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:26.765924 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:26.766347 (+ 423us) server_negotiation.cc:299] Negotiation successful
0806 04:54:26.766682 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":283,"threads_started":1}
I20260806 04:54:26.769376 28027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2ee6154f6844989945d52dfaee3eea" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 43547 } }
I20260806 04:54:26.769786 28027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:26.769932 28027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c2ee6154f6844989945d52dfaee3eea, State: Initialized, Role: FOLLOWER
I20260806 04:54:26.770537 28027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2ee6154f6844989945d52dfaee3eea" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 43547 } }
I20260806 04:54:26.770803 28027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:54:26.770958 28027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:54:26.771142 28027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:26.771567 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:54:26.773973 28027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2ee6154f6844989945d52dfaee3eea" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 43547 } }
I20260806 04:54:26.774533 28027 leader_election.cc:302] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c2ee6154f6844989945d52dfaee3eea; no voters:
I20260806 04:54:26.775131 28027 leader_election.cc:288] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:54:26.775262 28032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:54:26.775902 28032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [term 1 LEADER]: Becoming Leader. State: Replica: 3c2ee6154f6844989945d52dfaee3eea, State: Running, Role: LEADER
I20260806 04:54:26.776366 28032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2ee6154f6844989945d52dfaee3eea" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 43547 } }
I20260806 04:54:26.777096 28027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:54:26.779189 28034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c2ee6154f6844989945d52dfaee3eea. Latest consensus state: current_term: 1 leader_uuid: "3c2ee6154f6844989945d52dfaee3eea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2ee6154f6844989945d52dfaee3eea" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 43547 } } }
I20260806 04:54:26.779273 28033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c2ee6154f6844989945d52dfaee3eea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2ee6154f6844989945d52dfaee3eea" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 43547 } } }
I20260806 04:54:26.779515 28034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [sys.catalog]: This master's current role is: LEADER
I20260806 04:54:26.779558 28033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea [sys.catalog]: This master's current role is: LEADER
I20260806 04:54:26.782289 28039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:54:26.786270 28039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:54:26.793335 28039 catalog_manager.cc:1347] Generated new cluster ID: 7ef3a792b786404fb6ed04e26f9b5d7f
I20260806 04:54:26.793454 28039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:54:26.804056 28039 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:54:26.805123 28039 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:54:26.810091 28039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c2ee6154f6844989945d52dfaee3eea: Generated new TSK 0
I20260806 04:54:26.810880 28039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:54:26.833513 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 27959
I20260806 04:54:26.834576 28022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:26.914664 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 27959
2026-08-06T04:54:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1757 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:54:27 dist-test-slave-1x4g krb5kdc[28055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:54:27 dist-test-slave-1x4g krb5kdc[28055](info): set up 2 sockets
Aug 06 04:54:27 dist-test-slave-1x4g krb5kdc[28055](info): commencing operation
krb5kdc: starting...
W20260806 04:54:27.916800 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:54:28 dist-test-slave-1x4g krb5kdc[28055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:54:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:54:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:54:28.126019 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:33801
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:33801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:54:28.425253 28068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:28.425547 28068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:28.425714 28068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:28.432466 28068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:54:28.432600 28068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:28.432704 28068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:28.432823 28068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:54:28.432915 28068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:54:28.442114 28068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:33801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:33801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:28.443289 28068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:28.444494 28068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:28.455258 28073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:28.455264 28076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:28.455470 28074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:28.456434 28068 server_base.cc:1034] running on GCE node
I20260806 04:54:28.457274 28068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:28.458515 28068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:28.459703 28068 hybrid_clock.cc:648] HybridClock initialized: now 1785992068459671 us; error 45 us; skew 500 ppm
I20260806 04:54:28.460033 28068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:28 dist-test-slave-1x4g krb5kdc[28055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992068, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:28.467605 28068 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:54:28.468894 28068 webserver.cc:460] Webserver started at http://127.13.148.190:44947/ using document root <none> and password file <none>
I20260806 04:54:28.469520 28068 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:28.469662 28068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:28.469954 28068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:54:28.472709 28068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "15bc082abb514d35b41f0b62f46efe9c"
format_stamp: "Formatted at 2026-08-06 04:54:28 on dist-test-slave-1x4g"
server_key: "e8f888b50f6942691d7d02c8fc38db08"
server_key_iv: "a871449f130177db939011c5979a32e7"
server_key_version: "encryptionkey@0"
I20260806 04:54:28.473469 28068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "15bc082abb514d35b41f0b62f46efe9c"
format_stamp: "Formatted at 2026-08-06 04:54:28 on dist-test-slave-1x4g"
server_key: "e8f888b50f6942691d7d02c8fc38db08"
server_key_iv: "a871449f130177db939011c5979a32e7"
server_key_version: "encryptionkey@0"
I20260806 04:54:28.478112 28068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 04:54:28.481123 28083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:28.482673 28068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:54:28.482882 28068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "15bc082abb514d35b41f0b62f46efe9c"
format_stamp: "Formatted at 2026-08-06 04:54:28 on dist-test-slave-1x4g"
server_key: "e8f888b50f6942691d7d02c8fc38db08"
server_key_iv: "a871449f130177db939011c5979a32e7"
server_key_version: "encryptionkey@0"
I20260806 04:54:28.483094 28068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:28.496670 28068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:28.501044 28068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:28.501332 28068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:28.517906 28068 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:33801
I20260806 04:54:28.517925 28135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:33801 every 8 connection(s)
I20260806 04:54:28.519143 28068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:54:28.522419 28136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:28.524501 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 28068
I20260806 04:54:28.524721 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:54:28.525422 13906 external_mini_cluster.cc:1428] Setting key c2d2a29f25436843375728e2d612f122
I20260806 04:54:28.533232 28136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c: Bootstrap starting.
I20260806 04:54:28.536573 28136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:28.537904 28136 log.cc:824] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:54:28 dist-test-slave-1x4g krb5kdc[28055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:28.540565 28136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c: No bootstrap required, opened a new log
I20260806 04:54:28.548102 28139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:28.527287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.13.148.190:33801)
0806 04:54:28.528001 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:28.528012 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:28.528047 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:54:28.529532 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:28.529551 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:28.529881 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:28.530177 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:28.531916 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:28.533128 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:28.534767 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:28.535426 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:28.535548 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:28.539901 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:28.539910 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:28.539927 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:28.539976 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:28.543835 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:28.544425 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:28.544432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:28.544440 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:28.544666 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:28.545074 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:28.545078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:28.545080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:28.545670 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:28.546117 (+ 447us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:28.546404 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 04:54:28.546813 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":334,"threads_started":1}
I20260806 04:54:28.550840 28136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } }
I20260806 04:54:28.551180 28136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:28.551258 28136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15bc082abb514d35b41f0b62f46efe9c, State: Initialized, Role: FOLLOWER
I20260806 04:54:28.551774 28136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } }
I20260806 04:54:28.552033 28136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:54:28.552160 28136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:54:28.552304 28136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:28.554286 28136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } }
I20260806 04:54:28.554708 28136 leader_election.cc:302] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15bc082abb514d35b41f0b62f46efe9c; no voters:
I20260806 04:54:28.555212 28136 leader_election.cc:288] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:54:28.555328 28141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:54:28.555948 28141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 1 LEADER]: Becoming Leader. State: Replica: 15bc082abb514d35b41f0b62f46efe9c, State: Running, Role: LEADER
I20260806 04:54:28.556371 28141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } }
I20260806 04:54:28.557019 28136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:54:28.559247 28142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15bc082abb514d35b41f0b62f46efe9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } } }
I20260806 04:54:28.559347 28143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15bc082abb514d35b41f0b62f46efe9c. Latest consensus state: current_term: 1 leader_uuid: "15bc082abb514d35b41f0b62f46efe9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } } }
I20260806 04:54:28.559593 28143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [sys.catalog]: This master's current role is: LEADER
I20260806 04:54:28.559593 28142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [sys.catalog]: This master's current role is: LEADER
I20260806 04:54:28.560480 28148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:54:28.570052 28148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:54:28.578806 28148 catalog_manager.cc:1347] Generated new cluster ID: cad856918965462aae1f859489525254
I20260806 04:54:28.578933 28148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:54:28.613332 28148 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:54:28.614506 28148 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:54:28.621147 28148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c: Generated new TSK 0
I20260806 04:54:28.622123 28148 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:54:28.678020 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33801
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:54:28.991086 28162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:28.991361 28162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:28.991544 28162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:28.998322 28162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:28.998425 28162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:28.998580 28162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:54:29.007903 28162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:29.009124 28162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:29.010501 28162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:29.022927 28171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:29.022967 28168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:29.023005 28167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:29.026003 28162 server_base.cc:1034] running on GCE node
I20260806 04:54:29.026526 28162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:29.027359 28162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:29.028564 28162 hybrid_clock.cc:648] HybridClock initialized: now 1785992069028528 us; error 34 us; skew 500 ppm
I20260806 04:54:29.028895 28162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:29 dist-test-slave-1x4g krb5kdc[28055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992069, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:29.031795 28162 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:54:29.033232 28162 webserver.cc:460] Webserver started at http://127.13.148.129:44723/ using document root <none> and password file <none>
I20260806 04:54:29.033877 28162 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:29.034013 28162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:29.034292 28162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:54:29.037135 28162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "8c4f17f308b84beb89456c164c129b6f"
format_stamp: "Formatted at 2026-08-06 04:54:29 on dist-test-slave-1x4g"
server_key: "d597d7b4a400b4d672e1a2bc597eb5a0"
server_key_iv: "f90f0ac3bb0539983afe673da6e519d0"
server_key_version: "encryptionkey@0"
I20260806 04:54:29.037901 28162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "8c4f17f308b84beb89456c164c129b6f"
format_stamp: "Formatted at 2026-08-06 04:54:29 on dist-test-slave-1x4g"
server_key: "d597d7b4a400b4d672e1a2bc597eb5a0"
server_key_iv: "f90f0ac3bb0539983afe673da6e519d0"
server_key_version: "encryptionkey@0"
I20260806 04:54:29.042845 28162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 04:54:29.045706 28178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:29.046806 28162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:54:29.046963 28162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "8c4f17f308b84beb89456c164c129b6f"
format_stamp: "Formatted at 2026-08-06 04:54:29 on dist-test-slave-1x4g"
server_key: "d597d7b4a400b4d672e1a2bc597eb5a0"
server_key_iv: "f90f0ac3bb0539983afe673da6e519d0"
server_key_version: "encryptionkey@0"
I20260806 04:54:29.047149 28162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:29.059248 28162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:29.062878 28162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:29.063127 28162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:29.063905 28162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:29.065035 28162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:54:29.065125 28162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:29.065243 28162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:54:29.065306 28162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:29.094810 28162 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:35111
I20260806 04:54:29.094847 28292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:35111 every 8 connection(s)
I20260806 04:54:29.095896 28162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:54:29.100462 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 28162
I20260806 04:54:29.100664 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:54:29.101356 13906 external_mini_cluster.cc:1428] Setting key ffbdfd9e8e2a9efc58cb889673549f8a
Aug 06 04:54:29 dist-test-slave-1x4g krb5kdc[28055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992069, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:29.115011 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:29.098205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:45035 (local address 127.13.148.190:33801)
0806 04:54:29.098580 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:29.098584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:29.099436 (+ 852us) server_negotiation.cc:407] Connection header received
0806 04:54:29.101383 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:29.101388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:29.101477 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:29.101587 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:29.103413 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:29.104113 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:29.105707 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:29.106079 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:29.106209 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:29.109431 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:29.109436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:29.109439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:29.109473 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:29.112215 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:29.112852 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:29.112859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:29.112864 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:29.112997 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:29.113359 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:29.113365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:29.113371 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:29.113713 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:29.113848 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:29.114368 (+ 520us) server_negotiation.cc:299] Negotiation successful
0806 04:54:29.114603 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":189,"threads_started":1}
I20260806 04:54:29.115478 28296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:29.098675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:45035)
0806 04:54:29.099284 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:29.099322 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:29.101121 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:29.101836 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:29.101844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:29.102529 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:29.103201 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:29.103228 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:29.104280 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:29.104285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:29.105567 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:29.105580 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:29.106288 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:29.106297 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:29.106541 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:29.107154 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:29.107172 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:29.109239 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:29.112352 (+ 3113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:29.112357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:29.112366 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:29.112715 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:29.113092 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:29.113095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:29.113097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:29.113253 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:29.113818 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:29.113823 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:29.114062 (+ 239us) client_negotiation.cc:769] Sending connection context
0806 04:54:29.114241 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 04:54:29.114552 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":238,"threads_started":1}
I20260806 04:54:29.116852 28293 heartbeater.cc:347] Connected to a master server at 127.13.148.190:33801
I20260806 04:54:29.117146 28293 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:29.117990 28293 heartbeater.cc:511] Master 127.13.148.190:33801 requested a full tablet report, sending...
I20260806 04:54:29.119971 28101 ts_manager.cc:194] Registered new tserver with Master: 8c4f17f308b84beb89456c164c129b6f (127.13.148.129:35111)
I20260806 04:54:29.121721 28101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:45035
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:54:29.153190 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33801
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:54:29.445906 28299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:29.446137 28299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:29.446218 28299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:29.452600 28299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:29.452713 28299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:29.452895 28299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:54:29.461561 28299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:29.462790 28299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:29.463955 28299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:29.475646 28307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:29.475611 28305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:29.475816 28304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:29.476418 28299 server_base.cc:1034] running on GCE node
I20260806 04:54:29.477052 28299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:29.477892 28299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:29.479069 28299 hybrid_clock.cc:648] HybridClock initialized: now 1785992069479038 us; error 26 us; skew 500 ppm
I20260806 04:54:29.479338 28299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:29 dist-test-slave-1x4g krb5kdc[28055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992069, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:29.482298 28299 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:54:29.483533 28299 webserver.cc:460] Webserver started at http://127.13.148.130:46675/ using document root <none> and password file <none>
I20260806 04:54:29.484113 28299 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:29.484248 28299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:29.484508 28299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:54:29.487104 28299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616"
format_stamp: "Formatted at 2026-08-06 04:54:29 on dist-test-slave-1x4g"
server_key: "c99165c24cd4023c723102d1687ae777"
server_key_iv: "6765f22b519876e3890b04b03a74f37d"
server_key_version: "encryptionkey@0"
I20260806 04:54:29.487775 28299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616"
format_stamp: "Formatted at 2026-08-06 04:54:29 on dist-test-slave-1x4g"
server_key: "c99165c24cd4023c723102d1687ae777"
server_key_iv: "6765f22b519876e3890b04b03a74f37d"
server_key_version: "encryptionkey@0"
I20260806 04:54:29.492230 28299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 04:54:29.495118 28314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:29.496220 28299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 04:54:29.496379 28299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616"
format_stamp: "Formatted at 2026-08-06 04:54:29 on dist-test-slave-1x4g"
server_key: "c99165c24cd4023c723102d1687ae777"
server_key_iv: "6765f22b519876e3890b04b03a74f37d"
server_key_version: "encryptionkey@0"
I20260806 04:54:29.496562 28299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:29.518244 28299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:29.522068 28299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:29.522328 28299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:29.523142 28299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:29.524362 28299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:54:29.524451 28299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:29.524592 28299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:54:29.524655 28299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:29.555001 28299 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:44303
I20260806 04:54:29.555045 28430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:44303 every 8 connection(s)
I20260806 04:54:29.556056 28299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:54:29.561745 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 28299
I20260806 04:54:29.562022 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:54:29.562494 13906 external_mini_cluster.cc:1428] Setting key e3bb4fe866fe2816581b28fb4250cd5d
Aug 06 04:54:29 dist-test-slave-1x4g krb5kdc[28055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992069, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:29.577099 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:29.558435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:50767 (local address 127.13.148.190:33801)
0806 04:54:29.558623 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:29.558628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:29.559568 (+ 940us) server_negotiation.cc:407] Connection header received
0806 04:54:29.561389 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:29.561395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:29.561535 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:29.561652 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:29.563191 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:29.564127 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:29.565736 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:29.566420 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:29.566551 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:29.570429 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:29.570437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:29.570443 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:29.570482 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:29.573952 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:29.574896 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:29.574901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:29.574903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:29.575032 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:29.575477 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:29.575480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:29.575481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:29.575704 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:29.575815 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:29.576555 (+ 740us) server_negotiation.cc:299] Negotiation successful
0806 04:54:29.576838 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:54:29.578394 28433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:29.558823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:50767)
0806 04:54:29.559432 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:29.559465 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:29.561083 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:29.561983 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:29.562001 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:29.562497 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:29.563035 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:29.563052 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:29.564275 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:29.564279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:29.565564 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:29.565579 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:29.566711 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:29.566720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:29.566975 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:29.568284 (+ 1309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:29.568308 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:29.570227 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:29.574127 (+ 3900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:29.574135 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:29.574152 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:29.574694 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:29.575143 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:29.575149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:29.575154 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:29.575374 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:29.575813 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:29.575822 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:29.576206 (+ 384us) client_negotiation.cc:769] Sending connection context
0806 04:54:29.576469 (+ 263us) client_negotiation.cc:240] Negotiation successful
0806 04:54:29.577066 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":237,"threads_started":1}
I20260806 04:54:29.579691 28431 heartbeater.cc:347] Connected to a master server at 127.13.148.190:33801
I20260806 04:54:29.579962 28431 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:29.580591 28431 heartbeater.cc:511] Master 127.13.148.190:33801 requested a full tablet report, sending...
I20260806 04:54:29.582000 28101 ts_manager.cc:194] Registered new tserver with Master: 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303)
I20260806 04:54:29.582732 28101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:50767
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
I20260806 04:54:29.616080 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33801
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:54:29.918601 28436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:29.918825 28436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:29.918905 28436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:29.925508 28436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:29.925622 28436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:29.925776 28436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:54:29.934190 28436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:29.935328 28436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:29.936564 28436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:29.948551 28441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:29.948652 28443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:29.948830 28445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:29.949540 28436 server_base.cc:1034] running on GCE node
I20260806 04:54:29.950052 28436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:29.950842 28436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:29.952098 28436 hybrid_clock.cc:648] HybridClock initialized: now 1785992069951994 us; error 88 us; skew 500 ppm
I20260806 04:54:29.952399 28436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:29 dist-test-slave-1x4g krb5kdc[28055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992069, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:29.955454 28436 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:54:29.956899 28436 webserver.cc:460] Webserver started at http://127.13.148.131:44939/ using document root <none> and password file <none>
I20260806 04:54:29.957545 28436 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:29.957687 28436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:29.957995 28436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:54:29.960721 28436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "5a3c722ceec64f5389256e08c0ac1e76"
format_stamp: "Formatted at 2026-08-06 04:54:29 on dist-test-slave-1x4g"
server_key: "4865179d3a5ff09fa962b58e13c6dabd"
server_key_iv: "9c35e8ae00029eec65be4166855ecc72"
server_key_version: "encryptionkey@0"
I20260806 04:54:29.961628 28436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "5a3c722ceec64f5389256e08c0ac1e76"
format_stamp: "Formatted at 2026-08-06 04:54:29 on dist-test-slave-1x4g"
server_key: "4865179d3a5ff09fa962b58e13c6dabd"
server_key_iv: "9c35e8ae00029eec65be4166855ecc72"
server_key_version: "encryptionkey@0"
I20260806 04:54:29.966179 28436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:54:29.969224 28452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:29.970458 28436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260806 04:54:29.970623 28436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "5a3c722ceec64f5389256e08c0ac1e76"
format_stamp: "Formatted at 2026-08-06 04:54:29 on dist-test-slave-1x4g"
server_key: "4865179d3a5ff09fa962b58e13c6dabd"
server_key_iv: "9c35e8ae00029eec65be4166855ecc72"
server_key_version: "encryptionkey@0"
I20260806 04:54:29.970811 28436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:29.999140 28436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:30.003804 28436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:30.004112 28436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:30.005159 28436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:30.006803 28436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:54:30.006915 28436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:30.007066 28436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:54:30.007144 28436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:30.040161 28436 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:40157
I20260806 04:54:30.040208 28567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:40157 every 8 connection(s)
I20260806 04:54:30.041592 28436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:54:30.046489 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 28436
I20260806 04:54:30.046671 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:54:30.047134 13906 external_mini_cluster.cc:1428] Setting key 624f3db71075dab583489fa439ecf097
Aug 06 04:54:30 dist-test-slave-1x4g krb5kdc[28055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992069, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:30.060607 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:30.044037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:53031 (local address 127.13.148.190:33801)
0806 04:54:30.044227 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:30.044232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:30.045380 (+ 1148us) server_negotiation.cc:407] Connection header received
0806 04:54:30.047205 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:30.047210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:30.047301 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:30.047420 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:30.048937 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.049682 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.051358 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.051789 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.051925 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:30.055136 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:30.055141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:30.055144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:30.055175 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:30.058194 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:30.058790 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:30.058793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:30.058796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:30.058904 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:30.059241 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:30.059243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:30.059245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:30.059483 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:30.059584 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:30.060156 (+ 572us) server_negotiation.cc:299] Negotiation successful
0806 04:54:30.060316 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 04:54:30.061898 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:30.044530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:53031)
0806 04:54:30.045231 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:30.045272 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:30.046933 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:30.047608 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:30.047629 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:30.048128 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:30.048733 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:30.048749 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.049823 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.049828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:30.051181 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:30.051194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.052007 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.052017 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:30.052267 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:30.052967 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:30.052987 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:30.054957 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:30.058310 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:30.058316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:30.058326 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:30.058691 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:30.058992 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:30.058995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:30.058997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:30.059153 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:30.059579 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:30.059584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:30.059852 (+ 268us) client_negotiation.cc:769] Sending connection context
0806 04:54:30.060070 (+ 218us) client_negotiation.cc:240] Negotiation successful
0806 04:54:30.060480 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":308,"threads_started":1}
I20260806 04:54:30.063086 28568 heartbeater.cc:347] Connected to a master server at 127.13.148.190:33801
I20260806 04:54:30.063356 28568 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:30.064076 28568 heartbeater.cc:511] Master 127.13.148.190:33801 requested a full tablet report, sending...
I20260806 04:54:30.065418 28101 ts_manager.cc:194] Registered new tserver with Master: 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157)
I20260806 04:54:30.066152 28101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:53031
I20260806 04:54:30.074357 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:54:30.091514 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:30.078715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49740 (local address 127.13.148.190:33801)
0806 04:54:30.078861 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:30.078865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:30.079296 (+ 431us) server_negotiation.cc:407] Connection header received
0806 04:54:30.079484 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:30.079491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:30.079552 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:30.079649 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:30.081093 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.081801 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.083069 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.083433 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.083567 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:30.086387 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:30.086391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:30.086394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:30.086425 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:30.088896 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:30.089540 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:30.089547 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:30.089550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:30.089689 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:30.090089 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:30.090095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:30.090098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:30.090399 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:30.090647 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:30.090972 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 04:54:30.091196 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:54:30.095368 28101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:54:30.102284 28101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:54:30.125051 28293 heartbeater.cc:503] Master 127.13.148.190:33801 was elected leader, sending a full tablet report...
I20260806 04:54:30.138901 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:30.113782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:44303 (local address 127.0.0.1:40878)
0806 04:54:30.130737 (+ 16955us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:30.130762 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:30.130882 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:30.132608 (+ 1726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:30.132612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:30.132655 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:30.133038 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:30.133049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.134733 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.134738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:30.136446 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:30.136458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.138087 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.138093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:30.138225 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:30.138275 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 04:54:30.138357 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 04:54:30.138536 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16815,"mutex_wait_us":62,"thread_start_us":11363,"threads_started":1}
I20260806 04:54:30.138926 28581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:30.113498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:40157 (local address 127.0.0.1:59822)
0806 04:54:30.127137 (+ 13639us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:30.127183 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:30.127336 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:30.131422 (+ 4086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:30.131429 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:30.131460 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:30.131885 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:30.131900 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.133775 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.133783 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:30.135559 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:30.135570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.137663 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.137674 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:30.138132 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:30.138190 (+ 58us) client_negotiation.cc:769] Sending connection context
0806 04:54:30.138309 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 04:54:30.138503 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13376,"mutex_wait_us":214,"thread_start_us":18854,"threads_started":1}
I20260806 04:54:30.139722 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:30.113998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.13.148.130:44303)
0806 04:54:30.131483 (+ 17485us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:30.131494 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:30.131526 (+ 32us) server_negotiation.cc:407] Connection header received
0806 04:54:30.131630 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:30.131641 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:30.131880 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:30.132112 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:30.133219 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.134606 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.136609 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.137871 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.138022 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:30.138165 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:30.139385 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0806 04:54:30.139508 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17295,"thread_start_us":17133,"threads_started":1}
I20260806 04:54:30.140365 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:30.113808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.13.148.131:40157)
0806 04:54:30.130080 (+ 16272us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:30.130090 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:30.130123 (+ 33us) server_negotiation.cc:407] Connection header received
0806 04:54:30.130217 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:30.130226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:30.130456 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:30.131426 (+ 970us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:30.132124 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.133620 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.135704 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.137179 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.137359 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:30.137539 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:30.139933 (+ 2394us) server_negotiation.cc:299] Negotiation successful
0806 04:54:30.140075 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16089,"thread_start_us":16345,"threads_started":1}
I20260806 04:54:30.140388 28584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:30.115004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:35111 (local address 127.0.0.1:37670)
0806 04:54:30.132665 (+ 17661us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:30.132685 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:30.132828 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:30.133604 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:30.133608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:30.133639 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:30.134155 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:30.134167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.135871 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.135875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:30.138001 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:30.138012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.139685 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.139690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:30.139840 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:30.139874 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 04:54:30.139966 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 04:54:30.140070 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17513,"mutex_wait_us":39,"thread_start_us":17324,"threads_started":1}
I20260806 04:54:30.141083 28585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:30.115712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.13.148.129:35111)
0806 04:54:30.132188 (+ 16476us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:30.132196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:30.132809 (+ 613us) server_negotiation.cc:407] Connection header received
0806 04:54:30.132953 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:30.132963 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:30.133166 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:30.133341 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:30.134356 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.135740 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.138335 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.139494 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.139690 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:30.139839 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:30.140576 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 04:54:30.140729 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16283,"thread_start_us":16104,"threads_started":1}
I20260806 04:54:30.143558 28365 tablet_service.cc:1467] Processing CreateTablet for tablet 92af4dce63704193ba42faec982b2352 (DEFAULT_TABLE table=test-table [id=840b8eec35cb499490b14c301636bbdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:54:30.143743 28502 tablet_service.cc:1467] Processing CreateTablet for tablet 92af4dce63704193ba42faec982b2352 (DEFAULT_TABLE table=test-table [id=840b8eec35cb499490b14c301636bbdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:54:30.144759 28365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92af4dce63704193ba42faec982b2352. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:30.144341 28227 tablet_service.cc:1467] Processing CreateTablet for tablet 92af4dce63704193ba42faec982b2352 (DEFAULT_TABLE table=test-table [id=840b8eec35cb499490b14c301636bbdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:54:30.145072 28502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92af4dce63704193ba42faec982b2352. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:30.145504 28227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92af4dce63704193ba42faec982b2352. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:30.156164 28586 tablet_bootstrap.cc:492] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616: Bootstrap starting.
I20260806 04:54:30.158552 28587 tablet_bootstrap.cc:492] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f: Bootstrap starting.
I20260806 04:54:30.159646 28586 tablet_bootstrap.cc:654] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:30.160830 28586 log.cc:824] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:30.161468 28588 tablet_bootstrap.cc:492] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76: Bootstrap starting.
I20260806 04:54:30.161959 28587 tablet_bootstrap.cc:654] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:30.163439 28587 log.cc:824] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:30.164377 28588 tablet_bootstrap.cc:654] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:30.165604 28588 log.cc:824] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:30.166494 28587 tablet_bootstrap.cc:492] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f: No bootstrap required, opened a new log
I20260806 04:54:30.166649 28586 tablet_bootstrap.cc:492] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616: No bootstrap required, opened a new log
I20260806 04:54:30.166756 28587 ts_tablet_manager.cc:1397] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 04:54:30.166905 28586 ts_tablet_manager.cc:1397] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260806 04:54:30.168143 28588 tablet_bootstrap.cc:492] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76: No bootstrap required, opened a new log
I20260806 04:54:30.168362 28588 ts_tablet_manager.cc:1397] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260806 04:54:30.173725 28586 raft_consensus.cc:348] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } }
I20260806 04:54:30.174088 28586 raft_consensus.cc:374] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:30.174211 28586 raft_consensus.cc:729] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b5c265d9c7a49d9b9c1c1deee6a7616, State: Initialized, Role: FOLLOWER
I20260806 04:54:30.174736 28586 consensus_queue.cc:260] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } }
I20260806 04:54:30.175872 28431 heartbeater.cc:503] Master 127.13.148.190:33801 was elected leader, sending a full tablet report...
I20260806 04:54:30.176431 28586 ts_tablet_manager.cc:1428] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260806 04:54:30.176501 28587 raft_consensus.cc:348] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } }
I20260806 04:54:30.176965 28587 raft_consensus.cc:374] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:30.177066 28587 raft_consensus.cc:729] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c4f17f308b84beb89456c164c129b6f, State: Initialized, Role: FOLLOWER
I20260806 04:54:30.177631 28587 consensus_queue.cc:260] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } }
I20260806 04:54:30.177887 28588 raft_consensus.cc:348] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } }
I20260806 04:54:30.178295 28588 raft_consensus.cc:374] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:30.178395 28588 raft_consensus.cc:729] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3c722ceec64f5389256e08c0ac1e76, State: Initialized, Role: FOLLOWER
I20260806 04:54:30.179150 28587 ts_tablet_manager.cc:1428] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260806 04:54:30.179780 28588 consensus_queue.cc:260] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } }
I20260806 04:54:30.181175 28568 heartbeater.cc:503] Master 127.13.148.190:33801 was elected leader, sending a full tablet report...
I20260806 04:54:30.181334 28588 ts_tablet_manager.cc:1428] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260806 04:54:30.213846 28593 raft_consensus.cc:482] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:54:30.214102 28593 raft_consensus.cc:504] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } }
I20260806 04:54:30.215675 28593 leader_election.cc:288] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157), 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303)
I20260806 04:54:30.222533 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:30.215976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:52009 (local address 127.13.148.131:40157)
0806 04:54:30.216173 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:30.216179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:30.217160 (+ 981us) server_negotiation.cc:407] Connection header received
0806 04:54:30.217243 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:30.217252 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:30.217331 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:30.217423 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:30.218080 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.218862 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.220719 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.221574 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.221693 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:30.221794 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:30.222188 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 04:54:30.222306 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:54:30.222746 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:30.216117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:56413 (local address 127.13.148.130:44303)
0806 04:54:30.216322 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:30.216329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:30.217323 (+ 994us) server_negotiation.cc:407] Connection header received
0806 04:54:30.217490 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:30.217495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:30.217572 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:30.217688 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:30.218251 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.219002 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.220929 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.221754 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.221899 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:30.221971 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:30.222408 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 04:54:30.222533 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:54:30.222771 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:30.216102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:40157 (local address 127.13.148.129:52009)
0806 04:54:30.216956 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:30.216974 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:30.217083 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:30.217462 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:30.217466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:30.217490 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:30.217933 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:30.217954 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.219009 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.219014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:30.220575 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:30.220587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.221792 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.221801 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:30.222005 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:30.222044 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 04:54:30.222140 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 04:54:30.222233 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":2,"thread_start_us":449,"threads_started":1}
I20260806 04:54:30.222851 28596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:30.216125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:44303 (local address 127.13.148.129:56413)
0806 04:54:30.217216 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:30.217232 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:30.217349 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:30.217709 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:30.217712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:30.217730 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:30.218087 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:30.218097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.219139 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.219142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:30.220764 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:30.220777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.222047 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.222052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:30.222254 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:30.222282 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 04:54:30.222368 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 04:54:30.222484 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":18,"thread_start_us":509,"threads_started":1}
I20260806 04:54:30.223675 28523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92af4dce63704193ba42faec982b2352" candidate_uuid: "8c4f17f308b84beb89456c164c129b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3c722ceec64f5389256e08c0ac1e76" is_pre_election: true
I20260806 04:54:30.223866 28385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92af4dce63704193ba42faec982b2352" candidate_uuid: "8c4f17f308b84beb89456c164c129b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" is_pre_election: true
I20260806 04:54:30.224206 28523 raft_consensus.cc:2454] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c4f17f308b84beb89456c164c129b6f in term 0.
I20260806 04:54:30.224385 28385 raft_consensus.cc:2454] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c4f17f308b84beb89456c164c129b6f in term 0.
I20260806 04:54:30.224908 28180 leader_election.cc:302] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a3c722ceec64f5389256e08c0ac1e76, 8c4f17f308b84beb89456c164c129b6f; no voters:
I20260806 04:54:30.225304 28593 raft_consensus.cc:2781] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:54:30.225430 28593 raft_consensus.cc:482] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:54:30.225526 28593 raft_consensus.cc:3037] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:30.227603 28593 raft_consensus.cc:504] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } }
I20260806 04:54:30.228191 28593 leader_election.cc:288] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 1 election: Requested vote from peers 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157), 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303)
I20260806 04:54:30.228595 28385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92af4dce63704193ba42faec982b2352" candidate_uuid: "8c4f17f308b84beb89456c164c129b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616"
I20260806 04:54:30.228595 28523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92af4dce63704193ba42faec982b2352" candidate_uuid: "8c4f17f308b84beb89456c164c129b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3c722ceec64f5389256e08c0ac1e76"
I20260806 04:54:30.228821 28523 raft_consensus.cc:3037] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:30.228804 28385 raft_consensus.cc:3037] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:30.231915 28523 raft_consensus.cc:2454] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c4f17f308b84beb89456c164c129b6f in term 1.
I20260806 04:54:30.231915 28385 raft_consensus.cc:2454] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c4f17f308b84beb89456c164c129b6f in term 1.
I20260806 04:54:30.232301 28182 leader_election.cc:302] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b5c265d9c7a49d9b9c1c1deee6a7616, 8c4f17f308b84beb89456c164c129b6f; no voters:
I20260806 04:54:30.232589 28593 raft_consensus.cc:2781] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:54:30.233014 28593 raft_consensus.cc:686] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [term 1 LEADER]: Becoming Leader. State: Replica: 8c4f17f308b84beb89456c164c129b6f, State: Running, Role: LEADER
I20260806 04:54:30.233436 28593 consensus_queue.cc:237] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } }
I20260806 04:54:30.238035 28100 catalog_manager.cc:5495] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c4f17f308b84beb89456c164c129b6f (127.13.148.129). New cstate: current_term: 1 leader_uuid: "8c4f17f308b84beb89456c164c129b6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } health_report { overall_health: HEALTHY } } }
I20260806 04:54:30.265993 28585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:30.260566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37672 (local address 127.13.148.129:35111)
0806 04:54:30.260748 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:30.260754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:30.260814 (+ 60us) server_negotiation.cc:407] Connection header received
0806 04:54:30.261091 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:30.261096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:30.261213 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:30.261356 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:54:30.261782 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.262484 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.263761 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.264182 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.264319 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:30.264905 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:54:30.265422 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:54:30.265591 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:30.265665 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 04:54:30.265740 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:54:30.273933 28385 raft_consensus.cc:1261] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 1 FOLLOWER]: Refusing update from remote peer 8c4f17f308b84beb89456c164c129b6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:54:30.273933 28523 raft_consensus.cc:1261] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76 [term 1 FOLLOWER]: Refusing update from remote peer 8c4f17f308b84beb89456c164c129b6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:54:30.274919 28593 consensus_queue.cc:1035] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:54:30.275177 28597 consensus_queue.cc:1035] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:54:30.286603 28600 mvcc.cc:205] Tried to move back new op lower bound from 7315423519830982656 to 7315423519679737856. Current Snapshot: MvccSnapshot[applied={T|T < 7315423519830982656}]
I20260806 04:54:30.301189 28100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:54:30.301445 28100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:54:30.304533 28100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c: Sending DeleteTablet for 3 replicas of tablet 92af4dce63704193ba42faec982b2352
I20260806 04:54:30.305634 28365 tablet_service.cc:1514] Processing DeleteTablet for tablet 92af4dce63704193ba42faec982b2352 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:54:30 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:40878
I20260806 04:54:30.305810 28502 tablet_service.cc:1514] Processing DeleteTablet for tablet 92af4dce63704193ba42faec982b2352 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:54:30 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:59822
I20260806 04:54:30.306434 28610 tablet_replica.cc:330] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616: stopping tablet replica
I20260806 04:54:30.306442 28227 tablet_service.cc:1514] Processing DeleteTablet for tablet 92af4dce63704193ba42faec982b2352 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:54:30 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:37670
I20260806 04:54:30.306831 28610 raft_consensus.cc:2229] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:54:30.307271 28610 raft_consensus.cc:2258] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:54:30.307297 28612 tablet_replica.cc:330] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f: stopping tablet replica
I20260806 04:54:30.307423 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 28162
I20260806 04:54:30.307957 28612 raft_consensus.cc:2229] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:54:30.308552 28612 raft_consensus.cc:2258] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:54:30.309504 28610 ts_tablet_manager.cc:1905] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:54:30.310528 28612 ts_tablet_manager.cc:1905] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:54:30.317157 28612 ts_tablet_manager.cc:1918] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:54:30.317317 28610 ts_tablet_manager.cc:1918] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:54:30.317378 28612 log.cc:1196] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/92af4dce63704193ba42faec982b2352
I20260806 04:54:30.317499 28610 log.cc:1196] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/92af4dce63704193ba42faec982b2352
I20260806 04:54:30.318069 28610 ts_tablet_manager.cc:1939] T 92af4dce63704193ba42faec982b2352 P 6b5c265d9c7a49d9b9c1c1deee6a7616: Deleting consensus metadata
I20260806 04:54:30.318111 28612 ts_tablet_manager.cc:1939] T 92af4dce63704193ba42faec982b2352 P 8c4f17f308b84beb89456c164c129b6f: Deleting consensus metadata
I20260806 04:54:30.319835 28086 catalog_manager.cc:4841] TS 8c4f17f308b84beb89456c164c129b6f (127.13.148.129:35111): tablet 92af4dce63704193ba42faec982b2352 (table test-table [id=840b8eec35cb499490b14c301636bbdd]) successfully deleted
I20260806 04:54:30.319947 28087 catalog_manager.cc:4841] TS 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303): tablet 92af4dce63704193ba42faec982b2352 (table test-table [id=840b8eec35cb499490b14c301636bbdd]) successfully deleted
I20260806 04:54:30.322196 28611 tablet_replica.cc:330] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76: stopping tablet replica
I20260806 04:54:30.322613 28611 raft_consensus.cc:2229] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:54:30.323107 28611 raft_consensus.cc:2258] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:54:30.324712 28611 ts_tablet_manager.cc:1905] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:54:30.330766 28611 ts_tablet_manager.cc:1918] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:54:30.330957 28611 log.cc:1196] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/92af4dce63704193ba42faec982b2352
I20260806 04:54:30.331413 28611 ts_tablet_manager.cc:1939] T 92af4dce63704193ba42faec982b2352 P 5a3c722ceec64f5389256e08c0ac1e76: Deleting consensus metadata
Aug 06 04:54:30 dist-test-slave-1x4g krb5kdc[28055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:54:30.332952 28085 catalog_manager.cc:4841] TS 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157): tablet 92af4dce63704193ba42faec982b2352 (table test-table [id=840b8eec35cb499490b14c301636bbdd]) successfully deleted
I20260806 04:54:30.338992 28585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:30.308530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.13.148.129:35111)
0806 04:54:30.308691 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:30.308699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:30.321468 (+ 12769us) server_negotiation.cc:407] Connection header received
0806 04:54:30.321719 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:30.321725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:30.321810 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:30.321973 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:30.323904 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.325012 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.326774 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.327386 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.327507 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:30.332663 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:30.332677 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:30.332695 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:30.332731 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:30.335941 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:30.336863 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:30.336877 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:30.336881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:30.337051 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:30.337484 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:30.337487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:30.337489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:30.337809 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:30.338042 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:30.338429 (+ 387us) server_negotiation.cc:299] Negotiation successful
0806 04:54:30.338590 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:54:30.339682 28288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:30.431609 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 28162
I20260806 04:54:30.450693 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 28299
Aug 06 04:54:30 dist-test-slave-1x4g krb5kdc[28055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:54:30.466173 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:30.451465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40884 (local address 127.13.148.130:44303)
0806 04:54:30.451657 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:30.451666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:30.451749 (+ 83us) server_negotiation.cc:407] Connection header received
0806 04:54:30.451968 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:30.451979 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:30.452043 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:30.452160 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:30.453489 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.454139 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.455274 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.455626 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.455748 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:30.459910 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:30.459921 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:30.459936 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:30.459972 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:30.463332 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:30.463969 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:30.463978 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:30.463981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:30.464112 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:30.464470 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:30.464475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:30.464478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:30.464991 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:30.465263 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:30.465617 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 04:54:30.465826 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 04:54:30.466789 28426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:30.547184 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 28299
I20260806 04:54:30.565264 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 28436
Aug 06 04:54:30 dist-test-slave-1x4g krb5kdc[28055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
I20260806 04:54:30.580709 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:30.565936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59834 (local address 127.13.148.131:40157)
0806 04:54:30.566084 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:30.566088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:30.566105 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:54:30.566249 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:30.566252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:30.566310 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:30.566383 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:30.567505 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.568131 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.569611 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:30.570188 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:30.570321 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:30.575092 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:30.575115 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:30.575130 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:30.575199 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:30.577926 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:30.578654 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:30.578663 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:30.578667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:30.578820 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:30.579203 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:30.579209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:30.579213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:30.579592 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:30.579870 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:30.580190 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 04:54:30.580408 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:54:30.581373 28563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:30.661386 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 28436
I20260806 04:54:30.678653 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 28068
I20260806 04:54:30.679652 28131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:30.783375 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 28068
I20260806 04:54:30.805867 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:33801
--webserver_interface=127.13.148.190
--webserver_port=44947
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:33801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:54:31.116261 28618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:31.116536 28618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:31.116748 28618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:31.123363 28618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:54:31.123461 28618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:31.123503 28618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:31.123579 28618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:54:31.123677 28618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:54:31.132459 28618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:33801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:33801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=44947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:31.133611 28618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:31.134773 28618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:31.157089 28627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:31.157173 28624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:31.157140 28625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:31.158041 28618 server_base.cc:1034] running on GCE node
I20260806 04:54:31.158771 28618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:31.160167 28618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:31.161360 28618 hybrid_clock.cc:648] HybridClock initialized: now 1785992071161311 us; error 36 us; skew 500 ppm
I20260806 04:54:31.161767 28618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:31 dist-test-slave-1x4g krb5kdc[28055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992071, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:31.165014 28618 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:54:31.166368 28618 webserver.cc:460] Webserver started at http://127.13.148.190:44947/ using document root <none> and password file <none>
I20260806 04:54:31.167003 28618 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:31.167109 28618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:31.172398 28618 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:54:31.175133 28634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:31.176378 28618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 04:54:31.176540 28618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "15bc082abb514d35b41f0b62f46efe9c"
format_stamp: "Formatted at 2026-08-06 04:54:28 on dist-test-slave-1x4g"
server_key: "e8f888b50f6942691d7d02c8fc38db08"
server_key_iv: "a871449f130177db939011c5979a32e7"
server_key_version: "encryptionkey@0"
I20260806 04:54:31.177765 28618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:31.202032 28618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:31.205626 28618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:31.205852 28618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:31.223371 28618 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:33801
I20260806 04:54:31.223408 28685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:33801 every 8 connection(s)
I20260806 04:54:31.224748 28618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:54:31.228847 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 28618
I20260806 04:54:31.229782 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:35111
--local_ip_for_outbound_sockets=127.13.148.129
--tserver_master_addrs=127.13.148.190:33801
--webserver_port=44723
--webserver_interface=127.13.148.129
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:54:31.232012 28686 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:54:31.236917 28686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c: Bootstrap starting.
I20260806 04:54:31.257154 28686 log.cc:824] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:31.269305 28686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:54:31.269850 28686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c: Bootstrap complete.
I20260806 04:54:31.278940 28686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } }
I20260806 04:54:31.279765 28686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15bc082abb514d35b41f0b62f46efe9c, State: Initialized, Role: FOLLOWER
I20260806 04:54:31.280408 28686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } }
I20260806 04:54:31.280645 28686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:54:31.280825 28686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:54:31.281080 28686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:54:31.284730 28686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } }
I20260806 04:54:31.285291 28686 leader_election.cc:302] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15bc082abb514d35b41f0b62f46efe9c; no voters:
I20260806 04:54:31.285926 28686 leader_election.cc:288] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 04:54:31.286051 28690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:54:31.288237 28686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:54:31.293046 28690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 2 LEADER]: Becoming Leader. State: Replica: 15bc082abb514d35b41f0b62f46efe9c, State: Running, Role: LEADER
I20260806 04:54:31.293582 28690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } }
I20260806 04:54:31.296988 28691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "15bc082abb514d35b41f0b62f46efe9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } } }
I20260806 04:54:31.297483 28691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [sys.catalog]: This master's current role is: LEADER
I20260806 04:54:31.300462 28692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15bc082abb514d35b41f0b62f46efe9c. Latest consensus state: current_term: 2 leader_uuid: "15bc082abb514d35b41f0b62f46efe9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } } }
I20260806 04:54:31.300827 28692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [sys.catalog]: This master's current role is: LEADER
I20260806 04:54:31.301160 28701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:54:31.311798 28701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:54:31.313165 28701 catalog_manager.cc:1259] Loaded cluster ID: cad856918965462aae1f859489525254
I20260806 04:54:31.313288 28701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:54:31.316172 28701 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 04:54:31.331776 28701 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 04:54:31.339637 28708 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 04:54:31.343734 28701 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:54:31.345337 28701 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c: Loaded TSK: 0
I20260806 04:54:31.346663 28701 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 04:54:31.553738 28688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:31.554016 28688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:31.554172 28688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:31.560633 28688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:31.560768 28688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:31.560958 28688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:54:31.569244 28688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:35111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=44723
--tserver_master_addrs=127.13.148.190:33801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:31.570441 28688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:31.571736 28688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:31.586236 28715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:31.586269 28714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:31.590145 28717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:31.590547 28688 server_base.cc:1034] running on GCE node
I20260806 04:54:31.591300 28688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:31.592532 28688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:31.593772 28688 hybrid_clock.cc:648] HybridClock initialized: now 1785992071593705 us; error 47 us; skew 500 ppm
I20260806 04:54:31.594170 28688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:31 dist-test-slave-1x4g krb5kdc[28055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992071, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:31.597904 28688 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:54:31.599239 28688 webserver.cc:460] Webserver started at http://127.13.148.129:44723/ using document root <none> and password file <none>
I20260806 04:54:31.599860 28688 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:31.599980 28688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:31.605075 28688 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:54:31.607573 28724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:31.608924 28688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:54:31.609092 28688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "8c4f17f308b84beb89456c164c129b6f"
format_stamp: "Formatted at 2026-08-06 04:54:29 on dist-test-slave-1x4g"
server_key: "d597d7b4a400b4d672e1a2bc597eb5a0"
server_key_iv: "f90f0ac3bb0539983afe673da6e519d0"
server_key_version: "encryptionkey@0"
I20260806 04:54:31.609948 28688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:31.621980 28688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:31.625767 28688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:31.626006 28688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:31.626825 28688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:31.628037 28688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:54:31.628149 28688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:31.628252 28688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:54:31.628326 28688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:31.661458 28688 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:35111
I20260806 04:54:31.661514 28837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:35111 every 8 connection(s)
I20260806 04:54:31.662886 28688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:54:31.665505 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 28688
I20260806 04:54:31.666198 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:44303
--local_ip_for_outbound_sockets=127.13.148.130
--tserver_master_addrs=127.13.148.190:33801
--webserver_port=46675
--webserver_interface=127.13.148.130
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:54:31 dist-test-slave-1x4g krb5kdc[28055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992071, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:31.688941 28841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:31.665720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:42015)
0806 04:54:31.669310 (+ 3590us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:31.669353 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:31.670953 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:31.672516 (+ 1563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:31.672523 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:31.673127 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:31.673640 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:31.673656 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:31.675512 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:31.675519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:31.677617 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:31.677635 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:31.678636 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:31.678643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:31.678930 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:31.679600 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:31.679629 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:31.682158 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:31.685157 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:31.685165 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:31.685177 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:31.685575 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:31.686071 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:31.686074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:31.686076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:31.686231 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:31.687129 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:31.687134 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:31.687372 (+ 238us) client_negotiation.cc:769] Sending connection context
0806 04:54:31.687568 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 04:54:31.687885 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3334,"thread_start_us":3144,"threads_started":1}
I20260806 04:54:31.689491 28840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:31.665591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:42015 (local address 127.13.148.190:33801)
0806 04:54:31.666285 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:31.666300 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:31.669499 (+ 3199us) server_negotiation.cc:407] Connection header received
0806 04:54:31.671547 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:31.671567 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:31.672006 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:31.672405 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:31.673850 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:31.675360 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:31.677762 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:31.678387 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:31.678538 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:31.682430 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:31.682436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:31.682448 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:31.682484 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:31.685034 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:31.685740 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:31.685750 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:31.685759 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:31.685959 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:31.686344 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:31.686351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:31.686357 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:31.687000 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:31.687203 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:31.687831 (+ 628us) server_negotiation.cc:299] Negotiation successful
0806 04:54:31.688309 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":322,"threads_started":1}
I20260806 04:54:31.692021 28838 heartbeater.cc:347] Connected to a master server at 127.13.148.190:33801
I20260806 04:54:31.692333 28838 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:31.693095 28838 heartbeater.cc:511] Master 127.13.148.190:33801 requested a full tablet report, sending...
I20260806 04:54:31.695027 28651 ts_manager.cc:194] Registered new tserver with Master: 8c4f17f308b84beb89456c164c129b6f (127.13.148.129:35111)
I20260806 04:54:31.696341 28651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:42015
W20260806 04:54:31.970432 28842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:31.970700 28842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:31.970846 28842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:31.977414 28842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:31.977542 28842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:31.977698 28842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:54:31.986230 28842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:44303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=46675
--tserver_master_addrs=127.13.148.190:33801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:31.987340 28842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:31.988471 28842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:32.000736 28850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:32.000794 28848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:32.000942 28847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:32.001468 28842 server_base.cc:1034] running on GCE node
I20260806 04:54:32.002045 28842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:32.002856 28842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:32.004061 28842 hybrid_clock.cc:648] HybridClock initialized: now 1785992072004025 us; error 59 us; skew 500 ppm
I20260806 04:54:32.004359 28842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:32 dist-test-slave-1x4g krb5kdc[28055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992072, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:32.007516 28842 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:54:32.008865 28842 webserver.cc:460] Webserver started at http://127.13.148.130:46675/ using document root <none> and password file <none>
I20260806 04:54:32.009477 28842 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:32.009624 28842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:32.014362 28842 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260806 04:54:32.016891 28858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:32.018049 28842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 04:54:32.018204 28842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616"
format_stamp: "Formatted at 2026-08-06 04:54:29 on dist-test-slave-1x4g"
server_key: "c99165c24cd4023c723102d1687ae777"
server_key_iv: "6765f22b519876e3890b04b03a74f37d"
server_key_version: "encryptionkey@0"
I20260806 04:54:32.019037 28842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:32.043092 28842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:32.046759 28842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:32.047029 28842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:32.048009 28842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:32.049273 28842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:54:32.049387 28842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:32.049507 28842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:54:32.049590 28842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:32.079495 28842 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:44303
I20260806 04:54:32.079568 28971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:44303 every 8 connection(s)
I20260806 04:54:32.080894 28842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:54:32.082528 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 28842
I20260806 04:54:32.083072 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:40157
--local_ip_for_outbound_sockets=127.13.148.131
--tserver_master_addrs=127.13.148.190:33801
--webserver_port=44939
--webserver_interface=127.13.148.131
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:54:32 dist-test-slave-1x4g krb5kdc[28055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992072, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:32.102685 28840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.084291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:60479 (local address 127.13.148.190:33801)
0806 04:54:32.084471 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:32.084477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:32.084751 (+ 274us) server_negotiation.cc:407] Connection header received
0806 04:54:32.087265 (+ 2514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:32.087269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:32.087334 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:32.087410 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:32.089328 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.090038 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.092416 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.093088 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.093210 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.096231 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:32.096239 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:32.096246 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:32.096284 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:32.099549 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:32.100241 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:32.100248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:32.100254 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:32.100396 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:32.100770 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:32.100795 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:32.100800 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:32.101186 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:32.101352 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:32.101983 (+ 631us) server_negotiation.cc:299] Negotiation successful
0806 04:54:32.102280 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:54:32.103190 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.083829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:60479)
0806 04:54:32.084579 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:32.084635 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:32.086994 (+ 2359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:32.087666 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:32.087678 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:32.088395 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:32.089150 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:32.089176 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.090218 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.090226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:32.092272 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:32.092284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.093267 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.093271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:32.093544 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.094191 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:32.094210 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:32.096046 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:32.099758 (+ 3712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:32.099764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:32.099774 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:32.100124 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:32.100499 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:32.100502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:32.100503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:32.100671 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:32.101324 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:32.101329 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:32.101620 (+ 291us) client_negotiation.cc:769] Sending connection context
0806 04:54:32.101838 (+ 218us) client_negotiation.cc:240] Negotiation successful
0806 04:54:32.102202 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":279,"threads_started":1}
I20260806 04:54:32.104521 28972 heartbeater.cc:347] Connected to a master server at 127.13.148.190:33801
I20260806 04:54:32.104852 28972 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:32.105700 28972 heartbeater.cc:511] Master 127.13.148.190:33801 requested a full tablet report, sending...
I20260806 04:54:32.107335 28651 ts_manager.cc:194] Registered new tserver with Master: 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303)
I20260806 04:54:32.108155 28651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:60479
W20260806 04:54:32.387681 28975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:32.387921 28975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:32.388001 28975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:32.395027 28975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:32.395138 28975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:32.395296 28975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:54:32.403995 28975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:40157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=44939
--tserver_master_addrs=127.13.148.190:33801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:32.405347 28975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:32.406543 28975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:32.418376 28980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:32.418421 28981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:32.418336 28983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:32.419569 28975 server_base.cc:1034] running on GCE node
I20260806 04:54:32.420138 28975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:32.421046 28975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:32.422318 28975 hybrid_clock.cc:648] HybridClock initialized: now 1785992072422246 us; error 54 us; skew 500 ppm
I20260806 04:54:32.422670 28975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:32 dist-test-slave-1x4g krb5kdc[28055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992072, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:32.425601 28975 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:54:32.426851 28975 webserver.cc:460] Webserver started at http://127.13.148.131:44939/ using document root <none> and password file <none>
I20260806 04:54:32.427419 28975 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:32.427527 28975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:32.432308 28975 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:54:32.434631 28990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:32.435814 28975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:54:32.435977 28975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "5a3c722ceec64f5389256e08c0ac1e76"
format_stamp: "Formatted at 2026-08-06 04:54:29 on dist-test-slave-1x4g"
server_key: "4865179d3a5ff09fa962b58e13c6dabd"
server_key_iv: "9c35e8ae00029eec65be4166855ecc72"
server_key_version: "encryptionkey@0"
I20260806 04:54:32.436980 28975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:32.481611 28975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:32.485351 28975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:32.485594 28975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:32.486459 28975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:32.487607 28975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:54:32.487712 28975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:32.487838 28975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:54:32.487911 28975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:32.517565 28975 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:40157
I20260806 04:54:32.517648 29105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:40157 every 8 connection(s)
I20260806 04:54:32.518926 28975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:54:32.520237 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 28975
I20260806 04:54:32.535909 28840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.521258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49750 (local address 127.13.148.190:33801)
0806 04:54:32.521495 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:32.521501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:32.521906 (+ 405us) server_negotiation.cc:407] Connection header received
0806 04:54:32.522165 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:32.522171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:32.522275 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:32.522407 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:32.524728 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.525674 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.527088 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.527455 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.527573 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.530053 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:32.530059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:32.530063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:32.530102 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:32.533040 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:32.533815 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:32.533819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:32.533821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:32.534011 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:32.534462 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:32.534467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:32.534470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:32.534802 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:32.535089 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:32.535372 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 04:54:32.535583 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Aug 06 04:54:32 dist-test-slave-1x4g krb5kdc[28055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992072, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:32.543483 29109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.521621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:55655 (local address 127.13.148.190:33801)
0806 04:54:32.529146 (+ 7525us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:32.529150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:32.529162 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:54:32.529194 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:32.529197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:32.529238 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:32.529340 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:32.530860 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.532087 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.533553 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.534197 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.534319 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.538236 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:32.538242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:32.538244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:32.538269 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:32.540604 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:32.541294 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:32.541299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:32.541303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:32.541421 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:32.541874 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:32.541879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:32.541882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:32.542142 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:32.542309 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:32.542936 (+ 627us) server_negotiation.cc:299] Negotiation successful
0806 04:54:32.543119 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7448,"thread_start_us":7338,"threads_started":1}
I20260806 04:54:32.544461 29110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.521945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:55655)
0806 04:54:32.525300 (+ 3355us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:32.525361 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:32.528154 (+ 2793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:32.529569 (+ 1415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:32.529577 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:32.530054 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:32.530653 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:32.530680 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.532258 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.532261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:32.533419 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:32.533430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.534457 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.534465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:32.534755 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.535419 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:32.535442 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:32.538042 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:32.540725 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:32.540733 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:32.540745 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:32.541166 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:32.541541 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:32.541545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:32.541549 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:32.541750 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:32.542287 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:32.542294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:32.542597 (+ 303us) client_negotiation.cc:769] Sending connection context
0806 04:54:32.542843 (+ 246us) client_negotiation.cc:240] Negotiation successful
0806 04:54:32.543315 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3011,"thread_start_us":2747,"threads_started":1}
I20260806 04:54:32.545634 29106 heartbeater.cc:347] Connected to a master server at 127.13.148.190:33801
I20260806 04:54:32.545863 29106 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:32.546443 29106 heartbeater.cc:511] Master 127.13.148.190:33801 requested a full tablet report, sending...
I20260806 04:54:32.547598 28651 ts_manager.cc:194] Registered new tserver with Master: 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157)
I20260806 04:54:32.548252 28651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:55655
I20260806 04:54:32.550457 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:54:32.556279 29109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.551311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.13.148.190:33801)
0806 04:54:32.551494 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:32.551498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:32.551998 (+ 500us) server_negotiation.cc:407] Connection header received
0806 04:54:32.552159 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:32.552162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:32.552310 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:32.552406 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:54:32.552742 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.553365 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.554769 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.555098 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.555180 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.555542 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:54:32.555900 (+ 358us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:54:32.556006 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:32.556071 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 04:54:32.556118 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 04:54:32.557117 28651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:54:32.563519 28651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:54:32.599136 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.573670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59844 (local address 127.13.148.131:40157)
0806 04:54:32.591090 (+ 17420us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:32.591101 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:32.591203 (+ 102us) server_negotiation.cc:407] Connection header received
0806 04:54:32.591311 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:32.591320 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:32.591590 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:32.592369 (+ 779us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:32.592628 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.594197 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.596578 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.597968 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.598137 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.598291 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:32.598610 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 04:54:32.598762 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17233,"thread_start_us":17084,"threads_started":1}
I20260806 04:54:32.599232 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.573501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:35111 (local address 127.0.0.1:37702)
0806 04:54:32.589645 (+ 16144us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:32.589661 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:32.589791 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:32.592299 (+ 2508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:32.592305 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:32.592328 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:32.592729 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:32.592743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.594382 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.594386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:32.596524 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:32.596535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.598384 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.598389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:32.598563 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.598587 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 04:54:32.598634 (+ 47us) client_negotiation.cc:240] Negotiation successful
0806 04:54:32.598715 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16002,"mutex_wait_us":60,"thread_start_us":11744,"threads_started":1}
I20260806 04:54:32.599817 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.574665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40896 (local address 127.13.148.130:44303)
0806 04:54:32.590580 (+ 15915us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:32.590592 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:32.591494 (+ 902us) server_negotiation.cc:407] Connection header received
0806 04:54:32.591836 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:32.591844 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:32.592075 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:32.592302 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:32.593253 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.594467 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.596405 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.597966 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.598139 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.598291 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:32.599505 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0806 04:54:32.599612 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15689,"thread_start_us":15552,"threads_started":1}
I20260806 04:54:32.600025 29120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.574005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:44303 (local address 127.0.0.1:40896)
0806 04:54:32.591380 (+ 17375us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:32.591397 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:32.591591 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:32.592583 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:32.592587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:32.592610 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:32.593047 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:32.593061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.594603 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.594609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:32.596278 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:32.596290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.598453 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.598457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:32.598615 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.598643 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 04:54:32.598719 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 04:54:32.598839 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17249,"mutex_wait_us":2,"thread_start_us":17118,"threads_started":1}
I20260806 04:54:32.600040 29117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.573383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:40157 (local address 127.0.0.1:59844)
0806 04:54:32.585756 (+ 12373us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:32.585811 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:32.586020 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:32.591817 (+ 5797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:32.591822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:32.591845 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:32.592135 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:32.592143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.594330 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.594334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:32.596420 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:32.596435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.598156 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.598161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:32.598360 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.598419 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 04:54:32.598542 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 04:54:32.598700 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12043,"mutex_wait_us":20,"thread_start_us":16489,"threads_started":1}
I20260806 04:54:32.600219 29119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.573990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37702 (local address 127.13.148.129:35111)
0806 04:54:32.591768 (+ 17778us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:32.591776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:32.591801 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:54:32.591883 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:32.591889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:32.592075 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:32.592302 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:32.592952 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.594242 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.596876 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.598154 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.599626 (+ 1472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.599740 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:32.599785 (+ 45us) server_negotiation.cc:299] Negotiation successful
0806 04:54:32.599924 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17622,"thread_start_us":17529,"threads_started":1}
I20260806 04:54:32.603365 29040 tablet_service.cc:1467] Processing CreateTablet for tablet 21a0938b9bfc4cbd91011682665c519d (DEFAULT_TABLE table=test-table [id=1a79da63bc6d44f996f891cde672f9e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:54:32.603585 28772 tablet_service.cc:1467] Processing CreateTablet for tablet 21a0938b9bfc4cbd91011682665c519d (DEFAULT_TABLE table=test-table [id=1a79da63bc6d44f996f891cde672f9e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:54:32.603585 28906 tablet_service.cc:1467] Processing CreateTablet for tablet 21a0938b9bfc4cbd91011682665c519d (DEFAULT_TABLE table=test-table [id=1a79da63bc6d44f996f891cde672f9e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:54:32.604653 29040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a0938b9bfc4cbd91011682665c519d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:32.604766 28772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a0938b9bfc4cbd91011682665c519d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:32.604766 28906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a0938b9bfc4cbd91011682665c519d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:32.616151 29122 tablet_bootstrap.cc:492] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76: Bootstrap starting.
I20260806 04:54:32.618316 29123 tablet_bootstrap.cc:492] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616: Bootstrap starting.
I20260806 04:54:32.619807 29124 tablet_bootstrap.cc:492] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f: Bootstrap starting.
I20260806 04:54:32.619882 29122 tablet_bootstrap.cc:654] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:32.621347 29122 log.cc:824] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:32.621522 29123 tablet_bootstrap.cc:654] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:32.622761 29123 log.cc:824] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:32.623080 29124 tablet_bootstrap.cc:654] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:32.624116 29124 log.cc:824] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:32.624403 29122 tablet_bootstrap.cc:492] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76: No bootstrap required, opened a new log
I20260806 04:54:32.624681 29122 ts_tablet_manager.cc:1397] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 04:54:32.627779 29124 tablet_bootstrap.cc:492] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f: No bootstrap required, opened a new log
I20260806 04:54:32.628100 29124 ts_tablet_manager.cc:1397] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260806 04:54:32.630692 29123 tablet_bootstrap.cc:492] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616: No bootstrap required, opened a new log
I20260806 04:54:32.630920 29123 ts_tablet_manager.cc:1397] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260806 04:54:32.635197 29122 raft_consensus.cc:348] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } }
I20260806 04:54:32.635566 29122 raft_consensus.cc:374] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:32.635691 29122 raft_consensus.cc:729] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3c722ceec64f5389256e08c0ac1e76, State: Initialized, Role: FOLLOWER
I20260806 04:54:32.636247 29122 consensus_queue.cc:260] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } }
I20260806 04:54:32.637364 29106 heartbeater.cc:503] Master 127.13.148.190:33801 was elected leader, sending a full tablet report...
I20260806 04:54:32.638096 29123 raft_consensus.cc:348] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } }
I20260806 04:54:32.638444 29123 raft_consensus.cc:374] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:32.638571 29123 raft_consensus.cc:729] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b5c265d9c7a49d9b9c1c1deee6a7616, State: Initialized, Role: FOLLOWER
I20260806 04:54:32.638963 29124 raft_consensus.cc:348] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } }
I20260806 04:54:32.639361 29122 ts_tablet_manager.cc:1428] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76: Time spent starting tablet: real 0.015s user 0.006s sys 0.006s
I20260806 04:54:32.639441 29124 raft_consensus.cc:374] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:32.639601 29124 raft_consensus.cc:729] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c4f17f308b84beb89456c164c129b6f, State: Initialized, Role: FOLLOWER
I20260806 04:54:32.639480 29123 consensus_queue.cc:260] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } }
I20260806 04:54:32.640170 29124 consensus_queue.cc:260] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } }
I20260806 04:54:32.640533 28972 heartbeater.cc:503] Master 127.13.148.190:33801 was elected leader, sending a full tablet report...
I20260806 04:54:32.641243 29123 ts_tablet_manager.cc:1428] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260806 04:54:32.642391 28838 heartbeater.cc:503] Master 127.13.148.190:33801 was elected leader, sending a full tablet report...
I20260806 04:54:32.642889 29124 ts_tablet_manager.cc:1428] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
I20260806 04:54:32.645394 29130 raft_consensus.cc:482] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:54:32.645604 29130 raft_consensus.cc:504] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } }
I20260806 04:54:32.646989 29130 leader_election.cc:288] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157), 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303)
I20260806 04:54:32.653967 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.647788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:44807 (local address 127.13.148.131:40157)
0806 04:54:32.647968 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:32.647973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:32.648059 (+ 86us) server_negotiation.cc:407] Connection header received
0806 04:54:32.648253 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:32.648256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:32.648322 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:32.648388 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:32.649415 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.650396 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.652255 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.653175 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.653275 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.653318 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:32.653655 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 04:54:32.653728 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:54:32.654135 29131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.647356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:40157 (local address 127.13.148.129:44807)
0806 04:54:32.647943 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:32.647973 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:32.648124 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:32.648432 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:32.648439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:32.648468 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:32.649267 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:32.649300 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.650508 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.650513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:32.652096 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:32.652109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.653354 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.653360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:32.653506 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.653539 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 04:54:32.653632 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 04:54:32.653743 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":2,"thread_start_us":311,"threads_started":1}
I20260806 04:54:32.654699 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.647412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:44303 (local address 127.13.148.129:44671)
0806 04:54:32.649226 (+ 1814us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:32.649242 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:32.649401 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:32.649709 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:32.649713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:32.649750 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:32.650079 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:32.650089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.651141 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.651145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:32.652684 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:32.652697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.653877 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.653883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:32.654143 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.654165 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:54:32.654306 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 04:54:32.654430 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1692,"thread_start_us":1527,"threads_started":1}
I20260806 04:54:32.654918 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.647898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:44671 (local address 127.13.148.130:44303)
0806 04:54:32.648082 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:32.648087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:32.649310 (+ 1223us) server_negotiation.cc:407] Connection header received
0806 04:54:32.649553 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:32.649556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:32.649605 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:32.649667 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:32.650198 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.650992 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.652856 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.653691 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.653809 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.653878 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:32.654361 (+ 483us) server_negotiation.cc:299] Negotiation successful
0806 04:54:32.654530 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:54:32.655145 29060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a0938b9bfc4cbd91011682665c519d" candidate_uuid: "8c4f17f308b84beb89456c164c129b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3c722ceec64f5389256e08c0ac1e76" is_pre_election: true
I20260806 04:54:32.655648 29060 raft_consensus.cc:2454] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c4f17f308b84beb89456c164c129b6f in term 0.
I20260806 04:54:32.655774 28926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a0938b9bfc4cbd91011682665c519d" candidate_uuid: "8c4f17f308b84beb89456c164c129b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" is_pre_election: true
I20260806 04:54:32.656193 28926 raft_consensus.cc:2454] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c4f17f308b84beb89456c164c129b6f in term 0.
I20260806 04:54:32.656266 28726 leader_election.cc:302] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a3c722ceec64f5389256e08c0ac1e76, 8c4f17f308b84beb89456c164c129b6f; no voters:
I20260806 04:54:32.656647 29130 raft_consensus.cc:2781] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:54:32.656879 29130 raft_consensus.cc:482] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:54:32.657001 29130 raft_consensus.cc:3037] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:32.658933 29130 raft_consensus.cc:504] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } }
I20260806 04:54:32.659476 29130 leader_election.cc:288] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 1 election: Requested vote from peers 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157), 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303)
I20260806 04:54:32.659866 29060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a0938b9bfc4cbd91011682665c519d" candidate_uuid: "8c4f17f308b84beb89456c164c129b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3c722ceec64f5389256e08c0ac1e76"
I20260806 04:54:32.659866 28926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a0938b9bfc4cbd91011682665c519d" candidate_uuid: "8c4f17f308b84beb89456c164c129b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616"
I20260806 04:54:32.660036 29060 raft_consensus.cc:3037] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:32.660036 28926 raft_consensus.cc:3037] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:32.663017 29060 raft_consensus.cc:2454] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c4f17f308b84beb89456c164c129b6f in term 1.
I20260806 04:54:32.663017 28926 raft_consensus.cc:2454] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c4f17f308b84beb89456c164c129b6f in term 1.
I20260806 04:54:32.663358 28728 leader_election.cc:302] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b5c265d9c7a49d9b9c1c1deee6a7616, 8c4f17f308b84beb89456c164c129b6f; no voters:
I20260806 04:54:32.663691 29130 raft_consensus.cc:2781] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:54:32.664167 29130 raft_consensus.cc:686] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 LEADER]: Becoming Leader. State: Replica: 8c4f17f308b84beb89456c164c129b6f, State: Running, Role: LEADER
W20260806 04:54:32.664814 28839 tablet.cc:2369] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:54:32.664683 29130 consensus_queue.cc:237] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } }
I20260806 04:54:32.669093 28651 catalog_manager.cc:5495] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c4f17f308b84beb89456c164c129b6f (127.13.148.129). New cstate: current_term: 1 leader_uuid: "8c4f17f308b84beb89456c164c129b6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } health_report { overall_health: UNKNOWN } } }
I20260806 04:54:32.694278 29119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.689596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37716 (local address 127.13.148.129:35111)
0806 04:54:32.689809 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:32.689815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:32.689831 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:54:32.690034 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:32.690038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:32.690100 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:32.690198 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:54:32.690478 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.691091 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.692381 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.692749 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.692905 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.693331 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:54:32.693730 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:54:32.693821 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:32.693892 (+ 71us) server_negotiation.cc:299] Negotiation successful
0806 04:54:32.693979 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:54:32.701386 28926 raft_consensus.cc:1261] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 1 FOLLOWER]: Refusing update from remote peer 8c4f17f308b84beb89456c164c129b6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:54:32.701390 29060 raft_consensus.cc:1261] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76 [term 1 FOLLOWER]: Refusing update from remote peer 8c4f17f308b84beb89456c164c129b6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:54:32.702334 29133 consensus_queue.cc:1035] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:54:32.702670 29130 consensus_queue.cc:1035] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:54:32.715112 29137 mvcc.cc:205] Tried to move back new op lower bound from 7315423529776545792 to 7315423529638875136. Current Snapshot: MvccSnapshot[applied={T|T < 7315423529776545792}]
I20260806 04:54:32.725582 28650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:54:32.725848 28650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:54:32.729195 28650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c: Sending DeleteTablet for 3 replicas of tablet 21a0938b9bfc4cbd91011682665c519d
I20260806 04:54:32.730209 28772 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a0938b9bfc4cbd91011682665c519d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:54:32 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:37702
I20260806 04:54:32.730202 29040 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a0938b9bfc4cbd91011682665c519d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:54:32 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:59844
I20260806 04:54:32.730445 28906 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a0938b9bfc4cbd91011682665c519d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:54:32 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:40896
I20260806 04:54:32.731315 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 28688
I20260806 04:54:32.747408 29148 tablet_replica.cc:330] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616: stopping tablet replica
I20260806 04:54:32.748008 29148 raft_consensus.cc:2229] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:54:32.748677 29148 raft_consensus.cc:2258] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:54:32.752707 29148 ts_tablet_manager.cc:1905] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:54:32.759485 29148 ts_tablet_manager.cc:1918] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:54:32.759688 29148 log.cc:1196] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/21a0938b9bfc4cbd91011682665c519d
I20260806 04:54:32.759999 29119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.732304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.13.148.129:35111)
0806 04:54:32.732987 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:32.732995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:32.733015 (+ 20us) server_negotiation.cc:407] Connection header received
0806 04:54:32.733075 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:32.733082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:32.733177 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:32.733284 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:32.735526 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.737160 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.739800 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.740597 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.740726 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.748792 (+ 8066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:32.748826 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:32.748847 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:32.748908 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:32.753768 (+ 4860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:32.754915 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:32.754927 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:32.754934 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:32.755178 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:32.756344 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:32.756351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:32.756356 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:32.756977 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:32.757410 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:32.759169 (+ 1759us) server_negotiation.cc:299] Negotiation successful
0806 04:54:32.759442 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528}
I20260806 04:54:32.760282 29148 ts_tablet_manager.cc:1939] T 21a0938b9bfc4cbd91011682665c519d P 6b5c265d9c7a49d9b9c1c1deee6a7616: Deleting consensus metadata
I20260806 04:54:32.762125 28638 catalog_manager.cc:4841] TS 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303): tablet 21a0938b9bfc4cbd91011682665c519d (table test-table [id=1a79da63bc6d44f996f891cde672f9e0]) successfully deleted
I20260806 04:54:32.762482 28833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:32.766749 29145 tablet_replica.cc:330] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76: stopping tablet replica
I20260806 04:54:32.767083 29145 raft_consensus.cc:2229] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:54:32.767431 29145 raft_consensus.cc:2258] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:54:32.769062 29145 ts_tablet_manager.cc:1905] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:54:32.773387 29145 ts_tablet_manager.cc:1918] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:54:32.773515 29145 log.cc:1196] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/21a0938b9bfc4cbd91011682665c519d
I20260806 04:54:32.773924 29145 ts_tablet_manager.cc:1939] T 21a0938b9bfc4cbd91011682665c519d P 5a3c722ceec64f5389256e08c0ac1e76: Deleting consensus metadata
I20260806 04:54:32.775195 28636 catalog_manager.cc:4841] TS 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157): tablet 21a0938b9bfc4cbd91011682665c519d (table test-table [id=1a79da63bc6d44f996f891cde672f9e0]) successfully deleted
I20260806 04:54:32.858034 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 28688
I20260806 04:54:32.858197 29147 tablet_replica.cc:330] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f: stopping tablet replica
W20260806 04:54:32.876947 28637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.129:35111 (error 108)
I20260806 04:54:32.877280 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 28842
I20260806 04:54:32.878798 29117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.878454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:35111 (local address 127.0.0.1:37730)
0806 04:54:32.878584 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:32.878672 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:35111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 04:54:32.879035 28637 catalog_manager.cc:4584] TS 8c4f17f308b84beb89456c164c129b6f (127.13.148.129:35111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21a0938b9bfc4cbd91011682665c519d: Network error: Client connection negotiation failed: client connection to 127.13.148.129:35111: connect: Connection refused (error 111)
I20260806 04:54:32.890486 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.878105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.13.148.130:44303)
0806 04:54:32.878273 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:32.878279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:32.878320 (+ 41us) server_negotiation.cc:407] Connection header received
0806 04:54:32.878515 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:32.878520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:32.878596 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:32.878716 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:32.880061 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.880821 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.882210 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.882618 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.882761 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.885083 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:32.885093 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:32.885105 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:32.885133 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:32.887517 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:32.888239 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:32.888249 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:32.888252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:32.888391 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:32.888755 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:32.888759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:32.888762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:32.889314 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:32.889633 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:32.889925 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 04:54:32.890155 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:54:32.891134 28967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:32.932565 29117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.932077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:35111 (local address 127.0.0.1:37736)
0806 04:54:32.932301 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:32.932405 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:35111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 04:54:32.933543 29117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.933183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:35111 (local address 127.0.0.1:37742)
0806 04:54:32.933324 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:32.933421 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:35111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 04:54:32.971257 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 28842
I20260806 04:54:32.989418 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 28975
I20260806 04:54:32.994005 29117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.993542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:35111 (local address 127.0.0.1:37748)
0806 04:54:32.993719 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:32.993815 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:35111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260806 04:54:32.994990 29117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.994612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:35111 (local address 127.0.0.1:37764)
0806 04:54:32.994802 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:32.994889 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:35111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 04:54:33.002153 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:32.990102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.13.148.131:40157)
0806 04:54:32.990269 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:32.990273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:32.990290 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:54:32.990394 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:32.990397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:32.990464 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:32.990535 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:32.991797 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.992606 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.993973 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:32.994511 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:32.994626 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:32.997436 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:32.997462 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:32.997477 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:32.997508 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:32.999891 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:33.000468 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:33.000474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:33.000476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:33.000603 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:33.000956 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:33.000959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:33.000961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:33.001320 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:33.001560 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:33.001774 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 04:54:33.001923 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:54:33.002772 29101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:33.075876 29117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:33.075311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:35111 (local address 127.0.0.1:37770)
0806 04:54:33.075538 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:33.075669 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:35111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 04:54:33.077036 29117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:33.076541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:35111 (local address 127.0.0.1:37778)
0806 04:54:33.076777 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:33.076896 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:35111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260806 04:54:33.083009 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 28975
I20260806 04:54:33.100018 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 28618
I20260806 04:54:33.100991 28681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:33.202682 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 28618
I20260806 04:54:33.225751 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:33801
--webserver_interface=127.13.148.190
--webserver_port=44947
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:33801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:54:33.530406 29154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:33.530646 29154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:33.530720 29154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:33.537057 29154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:54:33.537173 29154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:33.537276 29154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:33.537386 29154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:54:33.537473 29154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:54:33.546125 29154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:33801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:33801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=44947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:33.547190 29154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:33.548339 29154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:33.559311 29159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:33.559286 29163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:33.559278 29160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:33.560308 29154 server_base.cc:1034] running on GCE node
I20260806 04:54:33.561160 29154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:33.562541 29154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:33.563751 29154 hybrid_clock.cc:648] HybridClock initialized: now 1785992073563694 us; error 46 us; skew 500 ppm
I20260806 04:54:33.564061 29154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:33 dist-test-slave-1x4g krb5kdc[28055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992073, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:33.571136 29154 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:54:33.572513 29154 webserver.cc:460] Webserver started at http://127.13.148.190:44947/ using document root <none> and password file <none>
I20260806 04:54:33.573163 29154 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:33.573282 29154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:33.578287 29154 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:54:33.581746 29170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:33.583158 29154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:54:33.583312 29154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "15bc082abb514d35b41f0b62f46efe9c"
format_stamp: "Formatted at 2026-08-06 04:54:28 on dist-test-slave-1x4g"
server_key: "e8f888b50f6942691d7d02c8fc38db08"
server_key_iv: "a871449f130177db939011c5979a32e7"
server_key_version: "encryptionkey@0"
I20260806 04:54:33.584153 29154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:33.618940 29154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:33.622602 29154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:33.622876 29154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:33.640385 29154 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:33801
I20260806 04:54:33.640410 29221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:33801 every 8 connection(s)
I20260806 04:54:33.641803 29154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:54:33.648085 29222 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:54:33.648677 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 29154
I20260806 04:54:33.649603 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:35111
--local_ip_for_outbound_sockets=127.13.148.129
--tserver_master_addrs=127.13.148.190:33801
--webserver_port=44723
--webserver_interface=127.13.148.129
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:54:33.652714 29222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c: Bootstrap starting.
I20260806 04:54:33.673848 29222 log.cc:824] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:33.694515 29222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:54:33.695149 29222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c: Bootstrap complete.
I20260806 04:54:33.708022 29222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } }
I20260806 04:54:33.709435 29222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15bc082abb514d35b41f0b62f46efe9c, State: Initialized, Role: FOLLOWER
I20260806 04:54:33.710343 29222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } }
I20260806 04:54:33.710713 29222 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:54:33.710856 29222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:54:33.711090 29222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:54:33.717324 29222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } }
I20260806 04:54:33.718152 29222 leader_election.cc:302] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15bc082abb514d35b41f0b62f46efe9c; no voters:
I20260806 04:54:33.718952 29222 leader_election.cc:288] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 04:54:33.719125 29226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:54:33.721486 29222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:54:33.723292 29226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [term 3 LEADER]: Becoming Leader. State: Replica: 15bc082abb514d35b41f0b62f46efe9c, State: Running, Role: LEADER
I20260806 04:54:33.723867 29226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } }
I20260806 04:54:33.726622 29227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "15bc082abb514d35b41f0b62f46efe9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } } }
I20260806 04:54:33.726969 29227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [sys.catalog]: This master's current role is: LEADER
I20260806 04:54:33.726598 29228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15bc082abb514d35b41f0b62f46efe9c. Latest consensus state: current_term: 3 leader_uuid: "15bc082abb514d35b41f0b62f46efe9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bc082abb514d35b41f0b62f46efe9c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33801 } } }
I20260806 04:54:33.728197 29228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bc082abb514d35b41f0b62f46efe9c [sys.catalog]: This master's current role is: LEADER
I20260806 04:54:33.728698 29233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:54:33.738173 29233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:54:33.739879 29233 catalog_manager.cc:1259] Loaded cluster ID: cad856918965462aae1f859489525254
I20260806 04:54:33.740046 29233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 04:54:33.742733 29233 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7faefe13d9ad google::LogMessage::Fail() at ??:0
@ 0x7faefe14088a google::LogMessage::SendToLog() at ??:0
@ 0x7faefe13d4de google::LogMessage::Flush() at ??:0
@ 0x7faefe13ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7faf13b70512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7faf13b6d976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7faf13bf5610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7faf13bf5439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7faf00d89345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7faf00d8c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7faf00d8c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7faf00d67eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7faf040416db start_thread at ??:0
@ 0x7faefcfd761f clone at ??:0
W20260806 04:54:34.082062 29224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:34.082315 29224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:34.082422 29224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:34.088481 29224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:34.088568 29224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:34.088671 29224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:54:34.097080 29224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:35111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=44723
--tserver_master_addrs=127.13.148.190:33801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:34.098196 29224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:34.099516 29224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:34.111615 29263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:34.111581 29264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:34.111639 29266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:34.112533 29224 server_base.cc:1034] running on GCE node
I20260806 04:54:34.113157 29224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:34.113961 29224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:34.115165 29224 hybrid_clock.cc:648] HybridClock initialized: now 1785992074115130 us; error 35 us; skew 500 ppm
I20260806 04:54:34.115447 29224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:34 dist-test-slave-1x4g krb5kdc[28055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:34.118607 29224 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:54:34.119966 29224 webserver.cc:460] Webserver started at http://127.13.148.129:44723/ using document root <none> and password file <none>
I20260806 04:54:34.120604 29224 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:34.120747 29224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:34.125711 29224 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260806 04:54:34.128392 29274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:34.129592 29224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:54:34.129734 29224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "8c4f17f308b84beb89456c164c129b6f"
format_stamp: "Formatted at 2026-08-06 04:54:29 on dist-test-slave-1x4g"
server_key: "d597d7b4a400b4d672e1a2bc597eb5a0"
server_key_iv: "f90f0ac3bb0539983afe673da6e519d0"
server_key_version: "encryptionkey@0"
I20260806 04:54:34.130542 29224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:34.146965 29224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:34.150308 29224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:34.150549 29224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:34.151412 29224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:34.153053 29281 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260806 04:54:34.158990 29224 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 04:54:34.159112 29224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260806 04:54:34.159240 29224 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 04:54:34.162029 29224 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 04:54:34.162132 29224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.001s
I20260806 04:54:34.162465 29281 tablet_bootstrap.cc:492] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f: Bootstrap starting.
I20260806 04:54:34.186357 29281 log.cc:824] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:34.199188 29281 tablet_bootstrap.cc:492] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:54:34.199689 29281 tablet_bootstrap.cc:492] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f: Bootstrap complete.
I20260806 04:54:34.200991 29281 ts_tablet_manager.cc:1397] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f: Time spent bootstrapping tablet: real 0.039s user 0.031s sys 0.007s
I20260806 04:54:34.208204 29281 raft_consensus.cc:348] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } }
I20260806 04:54:34.209378 29281 raft_consensus.cc:729] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c4f17f308b84beb89456c164c129b6f, State: Initialized, Role: FOLLOWER
I20260806 04:54:34.209971 29224 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:35111
I20260806 04:54:34.210004 29389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:35111 every 8 connection(s)
I20260806 04:54:34.210187 29281 consensus_queue.cc:260] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } }
I20260806 04:54:34.211483 29224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:54:34.211730 29281 ts_tablet_manager.cc:1428] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260806 04:54:34.215651 29393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:34.214073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:34813)
0806 04:54:34.214715 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:34.214918 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":274,"threads_started":1}
W20260806 04:54:34.216271 29391 heartbeater.cc:644] Failed to heartbeat to 127.13.148.190:33801 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.190:33801: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
I20260806 04:54:34.217132 29393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:34.216748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:58669)
0806 04:54:34.216969 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:34.217040 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260806 04:54:34.217931 29393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:34.217599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:34497)
0806 04:54:34.217740 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:34.217824 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 04:54:34.218148 29391 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 04:54:34.218406 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 29224
I20260806 04:54:34.218997 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:44303
--local_ip_for_outbound_sockets=127.13.148.130
--tserver_master_addrs=127.13.148.190:33801
--webserver_port=46675
--webserver_interface=127.13.148.130
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:54:34.513003 29394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:34.513273 29394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:34.513432 29394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:34.519830 29394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:34.519954 29394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:34.520090 29394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:54:34.528352 29394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:44303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=46675
--tserver_master_addrs=127.13.148.190:33801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:34.529450 29394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:34.530745 29394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:34.542899 29399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:34.542927 29402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:34.542990 29400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:34.544008 29394 server_base.cc:1034] running on GCE node
I20260806 04:54:34.544476 29394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:34.545290 29394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:34.546504 29394 hybrid_clock.cc:648] HybridClock initialized: now 1785992074546465 us; error 47 us; skew 500 ppm
I20260806 04:54:34.546779 29394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:34 dist-test-slave-1x4g krb5kdc[28055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:34.549654 29394 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:54:34.550910 29394 webserver.cc:460] Webserver started at http://127.13.148.130:46675/ using document root <none> and password file <none>
I20260806 04:54:34.551529 29394 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:34.551673 29394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:34.556186 29394 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:54:34.558701 29409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:34.559865 29394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 04:54:34.560027 29394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616"
format_stamp: "Formatted at 2026-08-06 04:54:29 on dist-test-slave-1x4g"
server_key: "c99165c24cd4023c723102d1687ae777"
server_key_iv: "6765f22b519876e3890b04b03a74f37d"
server_key_version: "encryptionkey@0"
I20260806 04:54:34.560886 29394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:34.576557 29394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:34.580022 29394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:34.580250 29394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:34.581067 29394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:34.582271 29394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:54:34.582408 29394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:34.582542 29394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:54:34.582625 29394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:34.614408 29394 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:44303
I20260806 04:54:34.614480 29522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:44303 every 8 connection(s)
I20260806 04:54:34.615839 29394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:54:34.620069 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:34.618555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:32779)
0806 04:54:34.619171 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:34.619340 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":238,"threads_started":1}
W20260806 04:54:34.620693 29523 heartbeater.cc:644] Failed to heartbeat to 127.13.148.190:33801 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.190:33801: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
I20260806 04:54:34.621413 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:34.621114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:45415)
0806 04:54:34.621246 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:34.621310 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260806 04:54:34.621923 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 29394
I20260806 04:54:34.622347 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:34.621950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:34413)
0806 04:54:34.622111 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:34.622199 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 04:54:34.622598 29523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 04:54:34.622713 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:40157
--local_ip_for_outbound_sockets=127.13.148.131
--tserver_master_addrs=127.13.148.190:33801
--webserver_port=44939
--webserver_interface=127.13.148.131
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:54:34.926342 29526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:34.926578 29526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:34.926658 29526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:34.933022 29526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:34.933130 29526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:34.933293 29526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:54:34.941730 29526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:40157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=44939
--tserver_master_addrs=127.13.148.190:33801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:34.942889 29526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:34.944151 29526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:34.956067 29534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:34.956116 29531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:34.956148 29532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:34.956919 29526 server_base.cc:1034] running on GCE node
I20260806 04:54:34.957543 29526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:34.958454 29526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:34.959632 29526 hybrid_clock.cc:648] HybridClock initialized: now 1785992074959596 us; error 37 us; skew 500 ppm
I20260806 04:54:34.959918 29526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:34 dist-test-slave-1x4g krb5kdc[28055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:34.962898 29526 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:54:34.964172 29526 webserver.cc:460] Webserver started at http://127.13.148.131:44939/ using document root <none> and password file <none>
I20260806 04:54:34.964884 29526 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:34.965001 29526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:34.969440 29526 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:54:34.971829 29541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:34.973140 29526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:54:34.973307 29526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "5a3c722ceec64f5389256e08c0ac1e76"
format_stamp: "Formatted at 2026-08-06 04:54:29 on dist-test-slave-1x4g"
server_key: "4865179d3a5ff09fa962b58e13c6dabd"
server_key_iv: "9c35e8ae00029eec65be4166855ecc72"
server_key_version: "encryptionkey@0"
I20260806 04:54:34.974231 29526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:34.986057 29526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:34.989714 29526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:34.989959 29526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:34.990742 29526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:34.991926 29526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:54:34.992010 29526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:34.992167 29526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:54:34.992257 29526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:35.022704 29526 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:40157
I20260806 04:54:35.022783 29653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:40157 every 8 connection(s)
I20260806 04:54:35.024000 29526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:54:35.025750 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 29526
I20260806 04:54:35.028486 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:35.026729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:47427)
0806 04:54:35.027481 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:35.027694 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":344,"threads_started":1}
W20260806 04:54:35.029072 29654 heartbeater.cc:644] Failed to heartbeat to 127.13.148.190:33801 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.190:33801: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
I20260806 04:54:35.029783 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:35.029504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:54581)
0806 04:54:35.029628 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:35.029697 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 04:54:35.030530 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:35.030208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:46089)
0806 04:54:35.030340 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:35.030433 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 04:54:35.030742 29654 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 04:54:35.219321 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:35.218612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:43049)
0806 04:54:35.219029 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:35.219125 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":210,"threads_started":1}
I20260806 04:54:35.396588 29659 raft_consensus.cc:482] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:54:35.397007 29659 raft_consensus.cc:504] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } }
I20260806 04:54:35.398653 29659 leader_election.cc:288] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157), 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303)
Aug 06 04:54:35 dist-test-slave-1x4g krb5kdc[28055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
Aug 06 04:54:35 dist-test-slave-1x4g krb5kdc[28055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:54:35.424495 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:35.399088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:40157 (local address 127.13.148.129:53237)
0806 04:54:35.399267 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:35.399306 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:35.399911 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:35.403756 (+ 3845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:35.403763 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:35.404578 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:35.405612 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:35.405638 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:35.407743 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:35.407747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:35.409953 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:35.409968 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:35.411428 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:35.411434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:35.411769 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:35.412716 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:35.412753 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:35.415587 (+ 2834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:35.419962 (+ 4375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:35.419970 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:35.419985 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:35.420764 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:35.421409 (+ 645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:35.421414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:35.421418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:35.421733 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:35.422742 (+ 1009us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:35.422750 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:35.423191 (+ 441us) client_negotiation.cc:769] Sending connection context
0806 04:54:35.423522 (+ 331us) client_negotiation.cc:240] Negotiation successful
0806 04:54:35.424050 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":458,"spinlock_wait_cycles":18560}
I20260806 04:54:35.424574 29660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:35.399193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:49791 (local address 127.13.148.130:44303)
0806 04:54:35.400955 (+ 1762us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:35.400968 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:35.401436 (+ 468us) server_negotiation.cc:407] Connection header received
0806 04:54:35.403445 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:35.403453 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:35.403811 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:35.404113 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:35.405816 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:35.407537 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:35.410129 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:35.411138 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:35.411336 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:35.416052 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:35.416059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:35.416070 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:35.416110 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:35.419818 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:35.420999 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:35.421006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:35.421012 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:35.421261 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:35.421860 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:35.421865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:35.421869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:35.422588 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:35.422776 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:35.423795 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0806 04:54:35.424183 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1609,"thread_start_us":1477,"threads_started":1}
I20260806 04:54:35.424574 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:35.399174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:53237 (local address 127.13.148.131:40157)
0806 04:54:35.400719 (+ 1545us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:35.400734 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:35.400799 (+ 65us) server_negotiation.cc:407] Connection header received
0806 04:54:35.402867 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:35.402879 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:35.403331 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:35.403656 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:35.405816 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:35.407537 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:35.410129 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:35.411138 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:35.411329 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:35.415897 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:35.415934 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:35.415947 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:35.415983 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:35.419781 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:35.420992 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:35.421001 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:35.421008 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:35.421261 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:35.421860 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:35.421865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:35.421869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:35.422588 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:35.422776 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:35.423797 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0806 04:54:35.424183 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1370,"thread_start_us":1228,"threads_started":1}
I20260806 04:54:35.424515 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:35.399447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:44303 (local address 127.13.148.129:49791)
0806 04:54:35.401308 (+ 1861us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:35.401323 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:35.401431 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:35.404107 (+ 2676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:35.404111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:35.404646 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:35.405612 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:35.405638 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:35.407740 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:35.407745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:35.409953 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:35.409968 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:35.411397 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:35.411410 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:35.411768 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:35.412716 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:35.412754 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:35.415782 (+ 3028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:35.419962 (+ 4180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:35.419970 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:35.419985 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:35.420764 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:35.421409 (+ 645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:35.421414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:35.421418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:35.421733 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:35.422742 (+ 1009us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:35.422750 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:35.423191 (+ 441us) client_negotiation.cc:769] Sending connection context
0806 04:54:35.423522 (+ 331us) client_negotiation.cc:240] Negotiation successful
0806 04:54:35.424173 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1745,"mutex_wait_us":601,"spinlock_wait_cycles":54016,"thread_start_us":1576,"threads_started":1}
W20260806 04:54:35.427677 29278 leader_election.cc:341] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303): Not found: Tablet not found: 21a0938b9bfc4cbd91011682665c519d
W20260806 04:54:35.427893 29276 leader_election.cc:341] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157): Not found: Tablet not found: 21a0938b9bfc4cbd91011682665c519d
I20260806 04:54:35.428040 29276 leader_election.cc:302] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c4f17f308b84beb89456c164c129b6f; no voters: 5a3c722ceec64f5389256e08c0ac1e76, 6b5c265d9c7a49d9b9c1c1deee6a7616
I20260806 04:54:35.428416 29659 raft_consensus.cc:2726] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:54:35.623871 29663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:35.623118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:43629)
0806 04:54:35.623526 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:35.623657 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":195,"threads_started":1}
I20260806 04:54:36.031926 29664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:36.031244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:58151)
0806 04:54:36.031644 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:36.031748 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":212,"threads_started":1}
I20260806 04:54:36.220858 29665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:36.220156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:54955)
0806 04:54:36.220593 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:36.220668 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":256,"threads_started":1}
I20260806 04:54:36.625264 29666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:36.624623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:45581)
0806 04:54:36.625059 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:36.625128 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":266,"threads_started":1}
I20260806 04:54:37.033491 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:37.032776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:38021)
0806 04:54:37.033239 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:37.033313 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":228,"threads_started":1}
I20260806 04:54:37.222311 29668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:37.221625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:42639)
0806 04:54:37.222076 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:37.222154 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":219,"threads_started":1}
I20260806 04:54:37.263840 29669 raft_consensus.cc:482] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:54:37.264029 29669 raft_consensus.cc:504] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } }
I20260806 04:54:37.264695 29669 leader_election.cc:288] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157), 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303)
W20260806 04:54:37.265614 29276 leader_election.cc:341] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157): Not found: Tablet not found: 21a0938b9bfc4cbd91011682665c519d
W20260806 04:54:37.265787 29278 leader_election.cc:341] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303): Not found: Tablet not found: 21a0938b9bfc4cbd91011682665c519d
I20260806 04:54:37.265947 29278 leader_election.cc:302] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c4f17f308b84beb89456c164c129b6f; no voters: 5a3c722ceec64f5389256e08c0ac1e76, 6b5c265d9c7a49d9b9c1c1deee6a7616
I20260806 04:54:37.266208 29669 raft_consensus.cc:2726] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:54:37.626578 29670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:37.625978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:33099)
0806 04:54:37.626364 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:37.626431 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":214,"threads_started":1}
I20260806 04:54:38.035132 29671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:38.034299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:33339)
0806 04:54:38.034819 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:38.034916 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":307,"threads_started":1}
I20260806 04:54:38.223770 29672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:38.223168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:52867)
0806 04:54:38.223557 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:38.223628 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":214,"threads_started":1}
I20260806 04:54:38.628089 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:38.627483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:49003)
0806 04:54:38.627852 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:38.627921 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":190,"threads_started":1}
I20260806 04:54:39.036535 29674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:39.035884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:35863)
0806 04:54:39.036275 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:39.036353 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":196,"threads_started":1}
I20260806 04:54:39.225565 29675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:39.224768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:36343)
0806 04:54:39.225321 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:39.225392 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":262,"threads_started":1}
I20260806 04:54:39.629671 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:39.629007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:32889)
0806 04:54:39.629409 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:39.629496 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":216,"threads_started":1}
I20260806 04:54:40.038045 29677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:40.037395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:48575)
0806 04:54:40.037817 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:40.037900 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":218,"threads_started":1}
I20260806 04:54:40.226881 29678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:40.226289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:47829)
0806 04:54:40.226656 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:40.226737 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":190,"threads_started":1}
I20260806 04:54:40.227231 29679 raft_consensus.cc:482] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:54:40.227382 29679 raft_consensus.cc:504] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } }
I20260806 04:54:40.227933 29679 leader_election.cc:288] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157), 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303)
W20260806 04:54:40.228685 29276 leader_election.cc:341] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157): Not found: Tablet not found: 21a0938b9bfc4cbd91011682665c519d
W20260806 04:54:40.228847 29278 leader_election.cc:341] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303): Not found: Tablet not found: 21a0938b9bfc4cbd91011682665c519d
I20260806 04:54:40.228997 29278 leader_election.cc:302] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c4f17f308b84beb89456c164c129b6f; no voters: 5a3c722ceec64f5389256e08c0ac1e76, 6b5c265d9c7a49d9b9c1c1deee6a7616
I20260806 04:54:40.229223 29679 raft_consensus.cc:2726] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:54:40.631134 29680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:40.630508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:59451)
0806 04:54:40.630888 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:40.630966 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":205,"threads_started":1}
I20260806 04:54:41.039848 29681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:41.038908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:41289)
0806 04:54:41.039505 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:41.039614 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":280,"threads_started":1}
I20260806 04:54:41.228490 29682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:41.227759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:49811)
0806 04:54:41.228203 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:41.228289 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":251,"threads_started":1}
I20260806 04:54:41.632907 29683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:41.632101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:45489)
0806 04:54:41.632590 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:41.632664 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":233,"threads_started":1}
I20260806 04:54:42.041553 29684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:42.040890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:39955)
0806 04:54:42.041304 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:42.041373 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":191,"threads_started":1}
I20260806 04:54:42.230000 29685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:42.229345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:42783)
0806 04:54:42.229766 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:42.229842 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":219,"threads_started":1}
I20260806 04:54:42.634464 29686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:42.633778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:42459)
0806 04:54:42.634205 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:42.634280 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":197,"threads_started":1}
I20260806 04:54:43.043107 29687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:43.042421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:37609)
0806 04:54:43.042816 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:43.042892 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":215,"threads_started":1}
I20260806 04:54:43.231513 29688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:43.230753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:56507)
0806 04:54:43.231229 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:43.231331 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":262,"threads_started":1}
I20260806 04:54:43.635919 29689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:43.635283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:41881)
0806 04:54:43.635715 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:43.635783 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":237,"threads_started":1}
I20260806 04:54:44.044720 29690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:44.044073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:59081)
0806 04:54:44.044517 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:44.044588 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":225,"threads_started":1}
I20260806 04:54:44.233354 29691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:44.232498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:56401)
0806 04:54:44.233037 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:44.233154 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":288,"threads_started":1}
I20260806 04:54:44.637523 29692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:44.636866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:55735)
0806 04:54:44.637272 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:44.637354 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":215,"threads_started":1}
I20260806 04:54:45.046207 29693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:45.045513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:58817)
0806 04:54:45.045918 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:45.046022 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":217,"threads_started":1}
I20260806 04:54:45.131923 29694 raft_consensus.cc:482] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:54:45.132105 29694 raft_consensus.cc:504] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } }
I20260806 04:54:45.132977 29694 leader_election.cc:288] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157), 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303)
W20260806 04:54:45.133782 29278 leader_election.cc:341] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303): Not found: Tablet not found: 21a0938b9bfc4cbd91011682665c519d
W20260806 04:54:45.133981 29276 leader_election.cc:341] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157): Not found: Tablet not found: 21a0938b9bfc4cbd91011682665c519d
I20260806 04:54:45.134068 29276 leader_election.cc:302] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c4f17f308b84beb89456c164c129b6f; no voters: 5a3c722ceec64f5389256e08c0ac1e76, 6b5c265d9c7a49d9b9c1c1deee6a7616
I20260806 04:54:45.134248 29694 raft_consensus.cc:2726] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:54:45.234818 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:45.234079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:60795)
0806 04:54:45.234514 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:45.234623 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":240,"threads_started":1}
I20260806 04:54:45.639079 29696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:45.638424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:60341)
0806 04:54:45.638842 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:45.638923 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":203,"threads_started":1}
I20260806 04:54:46.047670 29697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:46.047076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:59119)
0806 04:54:46.047466 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:46.047536 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":198,"threads_started":1}
I20260806 04:54:46.236220 29698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:46.235553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:47459)
0806 04:54:46.236010 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:46.236081 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":284,"threads_started":1}
I20260806 04:54:46.640643 29699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:46.640016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:60359)
0806 04:54:46.640415 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:46.640487 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":217,"threads_started":1}
I20260806 04:54:46.701630 29700 raft_consensus.cc:482] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:54:46.701829 29700 raft_consensus.cc:504] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3c722ceec64f5389256e08c0ac1e76" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 40157 } } peers { permanent_uuid: "8c4f17f308b84beb89456c164c129b6f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 35111 } } peers { permanent_uuid: "6b5c265d9c7a49d9b9c1c1deee6a7616" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44303 } }
I20260806 04:54:46.702479 29700 leader_election.cc:288] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157), 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303)
W20260806 04:54:46.703428 29278 leader_election.cc:341] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b5c265d9c7a49d9b9c1c1deee6a7616 (127.13.148.130:44303): Not found: Tablet not found: 21a0938b9bfc4cbd91011682665c519d
W20260806 04:54:46.703652 29276 leader_election.cc:341] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a3c722ceec64f5389256e08c0ac1e76 (127.13.148.131:40157): Not found: Tablet not found: 21a0938b9bfc4cbd91011682665c519d
I20260806 04:54:46.703759 29276 leader_election.cc:302] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c4f17f308b84beb89456c164c129b6f; no voters: 5a3c722ceec64f5389256e08c0ac1e76, 6b5c265d9c7a49d9b9c1c1deee6a7616
I20260806 04:54:46.703946 29700 raft_consensus.cc:2726] T 21a0938b9bfc4cbd91011682665c519d P 8c4f17f308b84beb89456c164c129b6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:54:47.049279 29701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:47.048582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:35211)
0806 04:54:47.049052 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:47.049126 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":259,"threads_started":1}
I20260806 04:54:47.237707 29702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:47.237021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:60595)
0806 04:54:47.237466 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:47.237541 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":239,"threads_started":1}
I20260806 04:54:47.642196 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:47.641460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:45469)
0806 04:54:47.641941 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:47.642025 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":244,"threads_started":1}
I20260806 04:54:48.050676 29704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:48.050033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:33675)
0806 04:54:48.050422 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:48.050501 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":196,"threads_started":1}
I20260806 04:54:48.239212 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:48.238527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:49571)
0806 04:54:48.238945 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:48.239031 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":220,"threads_started":1}
I20260806 04:54:48.643843 29706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:48.643016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:47311)
0806 04:54:48.643527 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:48.643598 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":267,"threads_started":1}
I20260806 04:54:49.052201 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:49.051486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:45455)
0806 04:54:49.051920 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:49.052034 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":233,"threads_started":1}
I20260806 04:54:49.241204 29708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:49.240360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:56353)
0806 04:54:49.240887 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:49.240972 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":224,"threads_started":1}
I20260806 04:54:49.645658 29709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:49.644957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:43315)
0806 04:54:49.645365 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:49.645493 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":213,"threads_started":1}
I20260806 04:54:50.038132 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 29224
I20260806 04:54:50.054080 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:50.053339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:47865)
0806 04:54:50.053800 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:50.053932 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":223,"threads_started":1}
I20260806 04:54:50.243005 29712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:50.242215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.129:57937)
0806 04:54:50.242741 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:50.242829 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":283,"threads_started":1}
I20260806 04:54:50.647404 29713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:50.646706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.130:43141)
0806 04:54:50.647160 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:50.647242 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":226,"threads_started":1}
I20260806 04:54:50.917465 29710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:50.039304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.13.148.129:35111)
0806 04:54:50.039960 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:50.039970 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:50.905636 (+865666us) server_negotiation.cc:407] Connection header received
0806 04:54:50.905728 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:50.905735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:50.905972 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:50.906107 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:50.907448 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:50.908312 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:50.909364 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:50.909953 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:50.910047 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:50.912608 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:50.912618 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:50.912632 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:50.912668 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:50.915227 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:50.915811 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:50.915817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:50.915819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:50.915932 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:50.916218 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:50.916220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:50.916222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:50.916528 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:50.916764 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:50.916985 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 04:54:50.917186 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":329,"threads_started":1}
I20260806 04:54:50.918680 29385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:51.000721 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 29224
I20260806 04:54:51.020117 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 29394
I20260806 04:54:51.033878 29715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:51.020975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38988 (local address 127.13.148.130:44303)
0806 04:54:51.021522 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:51.021526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:51.021541 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:54:51.021593 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:51.021596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:51.021649 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:51.021730 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:51.023072 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:51.023880 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:51.025543 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:51.025916 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:51.025998 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:51.028713 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:51.028717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:51.028720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:51.028749 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:51.031295 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:51.031943 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:51.031947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:51.031949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:51.032081 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:51.032399 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:51.032402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:51.032404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:51.032669 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:51.033044 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:51.033284 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 04:54:51.033454 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":313,"threads_started":1}
I20260806 04:54:51.034528 29518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:51.055802 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:51.055100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33801 (local address 127.13.148.131:34973)
0806 04:54:51.055570 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:51.055647 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":250,"threads_started":1}
I20260806 04:54:51.111207 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 29394
I20260806 04:54:51.129732 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 29526
I20260806 04:54:51.142042 29718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:51.130481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.13.148.131:40157)
0806 04:54:51.130991 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:51.130995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:51.131012 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:54:51.131063 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:51.131066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:51.131122 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:51.131204 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:51.132425 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:51.133294 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:51.134377 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:51.134821 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:51.134905 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:51.137348 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:51.137352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:51.137356 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:51.137384 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:51.139808 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:51.140305 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:51.140308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:51.140310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:51.140427 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:51.140723 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:51.140726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:51.140728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:51.141021 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:51.141202 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:51.141481 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 04:54:51.141636 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":257,"threads_started":1}
I20260806 04:54:51.142808 29649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:51.217031 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 29526
2026-08-06T04:54:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24304 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:54:51 dist-test-slave-1x4g krb5kdc[29722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:54:51 dist-test-slave-1x4g krb5kdc[29722](info): set up 2 sockets
Aug 06 04:54:51 dist-test-slave-1x4g krb5kdc[29722](info): commencing operation
krb5kdc: starting...
W20260806 04:54:52.225169 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:54:52 dist-test-slave-1x4g krb5kdc[29722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:54:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:54:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:54:52.446419 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41047
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:41047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:54:52.759969 29735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:52.760227 29735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:52.760332 29735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:52.766983 29735 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 04:54:52.767089 29735 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 04:54:52.767131 29735 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 04:54:52.767177 29735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:54:52.767225 29735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:52.767294 29735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:52.767340 29735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:54:52.767400 29735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:54:52.776180 29735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:41047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:52.777331 29735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:52.778554 29735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:52.790115 29741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:52.790179 29743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:52.790228 29740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:52.790856 29735 server_base.cc:1034] running on GCE node
I20260806 04:54:52.791460 29735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:52.792675 29735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:52.793907 29735 hybrid_clock.cc:648] HybridClock initialized: now 1785992092793840 us; error 30 us; skew 500 ppm
I20260806 04:54:52.794199 29735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:52 dist-test-slave-1x4g krb5kdc[29722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992092, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:52.797235 29735 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:54:52.798540 29735 webserver.cc:460] Webserver started at http://127.13.148.190:40965/ using document root <none> and password file <none>
I20260806 04:54:52.799150 29735 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:52.799268 29735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:52.799549 29735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:54:52.802305 29735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "a79f52b5e4274220b876f5422483397c"
format_stamp: "Formatted at 2026-08-06 04:54:52 on dist-test-slave-1x4g"
server_key: "6e2608cab18140526aa8ee3c77fdd452"
server_key_iv: "26555c209b659eeaacfced78e4055c91"
server_key_version: "encryptionkey@0"
I20260806 04:54:52.803057 29735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "a79f52b5e4274220b876f5422483397c"
format_stamp: "Formatted at 2026-08-06 04:54:52 on dist-test-slave-1x4g"
server_key: "6e2608cab18140526aa8ee3c77fdd452"
server_key_iv: "26555c209b659eeaacfced78e4055c91"
server_key_version: "encryptionkey@0"
I20260806 04:54:52.807770 29735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260806 04:54:52.810945 29750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:52.812032 29735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:54:52.812188 29735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "a79f52b5e4274220b876f5422483397c"
format_stamp: "Formatted at 2026-08-06 04:54:52 on dist-test-slave-1x4g"
server_key: "6e2608cab18140526aa8ee3c77fdd452"
server_key_iv: "26555c209b659eeaacfced78e4055c91"
server_key_version: "encryptionkey@0"
I20260806 04:54:52.812381 29735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:52.837687 29735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:52.841547 29735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:52.841792 29735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:52.858839 29735 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:41047
I20260806 04:54:52.858850 29802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:41047 every 8 connection(s)
I20260806 04:54:52.860198 29735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:54:52.863660 29803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:52.865739 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 29735
I20260806 04:54:52.865957 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:54:52.866623 13906 external_mini_cluster.cc:1428] Setting key 440c22e09bab6a784082c4165dd7fe78
I20260806 04:54:52.875826 29803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Bootstrap starting.
I20260806 04:54:52.879144 29803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:52.880053 29803 log.cc:824] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:54:52 dist-test-slave-1x4g krb5kdc[29722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:52.882915 29803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: No bootstrap required, opened a new log
I20260806 04:54:52.890528 29806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:52.868566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54090 (local address 127.13.148.190:41047)
0806 04:54:52.869402 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:52.869417 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:52.869462 (+ 45us) server_negotiation.cc:407] Connection header received
0806 04:54:52.871536 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:52.871565 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:52.872047 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:52.872453 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:52.873854 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:52.875533 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:52.877109 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:52.878172 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:52.878339 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:52.882354 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:52.882360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:52.882372 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:52.882401 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:52.886176 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:52.886889 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:52.886897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:52.886904 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:52.887142 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:52.887501 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:52.887505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:52.887507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:52.888093 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:52.888435 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:52.888727 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 04:54:52.889175 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":371,"threads_started":1}
I20260806 04:54:52.892908 29803 raft_consensus.cc:348] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } }
I20260806 04:54:52.893268 29803 raft_consensus.cc:374] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:52.893396 29803 raft_consensus.cc:729] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a79f52b5e4274220b876f5422483397c, State: Initialized, Role: FOLLOWER
I20260806 04:54:52.893945 29803 consensus_queue.cc:260] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } }
I20260806 04:54:52.894170 29803 raft_consensus.cc:388] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:54:52.894320 29803 raft_consensus.cc:482] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:54:52.894488 29803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:52.896468 29803 raft_consensus.cc:504] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } }
I20260806 04:54:52.897066 29803 leader_election.cc:302] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a79f52b5e4274220b876f5422483397c; no voters:
I20260806 04:54:52.897662 29803 leader_election.cc:288] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:54:52.897758 29808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:54:52.898516 29808 raft_consensus.cc:686] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 1 LEADER]: Becoming Leader. State: Replica: a79f52b5e4274220b876f5422483397c, State: Running, Role: LEADER
I20260806 04:54:52.899065 29808 consensus_queue.cc:237] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } }
I20260806 04:54:52.899801 29803 sys_catalog.cc:564] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:54:52.901754 29809 sys_catalog.cc:455] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a79f52b5e4274220b876f5422483397c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } } }
I20260806 04:54:52.901719 29810 sys_catalog.cc:455] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a79f52b5e4274220b876f5422483397c. Latest consensus state: current_term: 1 leader_uuid: "a79f52b5e4274220b876f5422483397c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } } }
I20260806 04:54:52.902211 29810 sys_catalog.cc:458] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [sys.catalog]: This master's current role is: LEADER
I20260806 04:54:52.902213 29809 sys_catalog.cc:458] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [sys.catalog]: This master's current role is: LEADER
I20260806 04:54:52.905689 29813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:54:52.910090 29813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:54:52.918291 29813 catalog_manager.cc:1347] Generated new cluster ID: ce1261c06295411fbb437b0bce6b5f4d
I20260806 04:54:52.918398 29813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:54:52.941882 29813 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:54:52.943238 29813 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:54:52.950376 29813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Generated new TSK 0
I20260806 04:54:52.951093 29813 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:54:53.008579 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41047
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:54:53.312290 29829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:53.312546 29829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:53.312657 29829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:53.319321 29829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:53.319429 29829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:53.319535 29829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:54:53.327996 29829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:53.329005 29829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:53.330116 29829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:53.341753 29834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:53.341715 29835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:53.342054 29837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:53.342700 29829 server_base.cc:1034] running on GCE node
I20260806 04:54:53.343209 29829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:53.344094 29829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:53.345284 29829 hybrid_clock.cc:648] HybridClock initialized: now 1785992093345241 us; error 54 us; skew 500 ppm
I20260806 04:54:53.345577 29829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:53 dist-test-slave-1x4g krb5kdc[29722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992093, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:53.348480 29829 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:54:53.349781 29829 webserver.cc:460] Webserver started at http://127.13.148.129:40687/ using document root <none> and password file <none>
I20260806 04:54:53.350370 29829 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:53.350500 29829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:53.350792 29829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:54:53.353545 29829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "36d1b567085d45a080c2dda2c9bdb063"
format_stamp: "Formatted at 2026-08-06 04:54:53 on dist-test-slave-1x4g"
server_key: "8f32e550d2cbbd33a9648bfcb0825a8a"
server_key_iv: "0f95f0c46d3b4b86102bc3c2b2d216ff"
server_key_version: "encryptionkey@0"
I20260806 04:54:53.354261 29829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "36d1b567085d45a080c2dda2c9bdb063"
format_stamp: "Formatted at 2026-08-06 04:54:53 on dist-test-slave-1x4g"
server_key: "8f32e550d2cbbd33a9648bfcb0825a8a"
server_key_iv: "0f95f0c46d3b4b86102bc3c2b2d216ff"
server_key_version: "encryptionkey@0"
I20260806 04:54:53.359052 29829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:54:53.361876 29845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:53.362979 29829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:54:53.363133 29829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "36d1b567085d45a080c2dda2c9bdb063"
format_stamp: "Formatted at 2026-08-06 04:54:53 on dist-test-slave-1x4g"
server_key: "8f32e550d2cbbd33a9648bfcb0825a8a"
server_key_iv: "0f95f0c46d3b4b86102bc3c2b2d216ff"
server_key_version: "encryptionkey@0"
I20260806 04:54:53.363322 29829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:53.374022 29829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:53.382400 29829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:53.382692 29829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:53.383544 29829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:53.384707 29829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:54:53.384848 29829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:53.384986 29829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:54:53.385072 29829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:53.414013 29829 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:36149
I20260806 04:54:53.414058 29959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:36149 every 8 connection(s)
I20260806 04:54:53.415081 29829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:54:53.419335 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 29829
I20260806 04:54:53.419530 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:54:53.420356 13906 external_mini_cluster.cc:1428] Setting key a518cf7af8e19719834ea1d69aa870a0
Aug 06 04:54:53 dist-test-slave-1x4g krb5kdc[29722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992093, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:53.435739 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:53.417622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:48711 (local address 127.13.148.190:41047)
0806 04:54:53.418037 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:53.418041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:53.418919 (+ 878us) server_negotiation.cc:407] Connection header received
0806 04:54:53.421248 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:53.421253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:53.421353 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:53.421460 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:53.422881 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:53.424104 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:53.425796 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:53.426379 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:53.426526 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:53.429781 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:53.429788 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:53.429795 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:53.429838 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:53.432715 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:53.433666 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:53.433670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:53.433672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:53.433808 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:53.434261 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:53.434264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:53.434266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:53.434512 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:53.434611 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:53.435358 (+ 747us) server_negotiation.cc:299] Negotiation successful
0806 04:54:53.435507 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":221,"threads_started":1}
I20260806 04:54:53.437191 29963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:53.418159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:48711)
0806 04:54:53.418756 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:53.418793 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:53.420999 (+ 2206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:53.421648 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:53.421654 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:53.422193 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:53.422703 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:53.422721 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:53.424242 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:53.424245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:53.425638 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:53.425658 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:53.426623 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:53.426632 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:53.426878 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:53.427542 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:53.427563 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:53.429596 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:53.432903 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:53.432912 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:53.432927 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:53.433548 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:53.433920 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:53.433927 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:53.433933 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:53.434157 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:53.434622 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:53.434630 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:53.434992 (+ 362us) client_negotiation.cc:769] Sending connection context
0806 04:54:53.435283 (+ 291us) client_negotiation.cc:240] Negotiation successful
0806 04:54:53.435813 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":240,"threads_started":1}
I20260806 04:54:53.438709 29960 heartbeater.cc:347] Connected to a master server at 127.13.148.190:41047
I20260806 04:54:53.439009 29960 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:53.439689 29960 heartbeater.cc:511] Master 127.13.148.190:41047 requested a full tablet report, sending...
I20260806 04:54:53.441588 29767 ts_manager.cc:194] Registered new tserver with Master: 36d1b567085d45a080c2dda2c9bdb063 (127.13.148.129:36149)
I20260806 04:54:53.442907 29767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:48711
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:54:53.475798 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41047
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:54:53.781404 29966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:53.781641 29966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:53.781810 29966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:53.788301 29966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:53.788386 29966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:53.788519 29966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:54:53.798313 29966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:53.799999 29966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:53.801602 29966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:53.818588 29972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:53.818624 29971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:53.820983 29975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:53.821419 29966 server_base.cc:1034] running on GCE node
I20260806 04:54:53.822309 29966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:53.823565 29966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:53.824815 29966 hybrid_clock.cc:648] HybridClock initialized: now 1785992093824725 us; error 61 us; skew 500 ppm
I20260806 04:54:53.825196 29966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:53 dist-test-slave-1x4g krb5kdc[29722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992093, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:53.829381 29966 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:54:53.831012 29966 webserver.cc:460] Webserver started at http://127.13.148.130:38069/ using document root <none> and password file <none>
I20260806 04:54:53.832010 29966 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:53.832149 29966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:53.832540 29966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:54:53.836148 29966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "e4fda8f8b55d45c9bef0073d6278d7f3"
format_stamp: "Formatted at 2026-08-06 04:54:53 on dist-test-slave-1x4g"
server_key: "1c47c0ebfec6bde4c167d911a546e37c"
server_key_iv: "8c571feac16eda73ab981e7544f9fc77"
server_key_version: "encryptionkey@0"
I20260806 04:54:53.837236 29966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "e4fda8f8b55d45c9bef0073d6278d7f3"
format_stamp: "Formatted at 2026-08-06 04:54:53 on dist-test-slave-1x4g"
server_key: "1c47c0ebfec6bde4c167d911a546e37c"
server_key_iv: "8c571feac16eda73ab981e7544f9fc77"
server_key_version: "encryptionkey@0"
I20260806 04:54:53.844267 29966 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260806 04:54:53.848917 29982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:53.850486 29966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260806 04:54:53.850646 29966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "e4fda8f8b55d45c9bef0073d6278d7f3"
format_stamp: "Formatted at 2026-08-06 04:54:53 on dist-test-slave-1x4g"
server_key: "1c47c0ebfec6bde4c167d911a546e37c"
server_key_iv: "8c571feac16eda73ab981e7544f9fc77"
server_key_version: "encryptionkey@0"
I20260806 04:54:53.850869 29966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:53.883201 29966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:53.886976 29966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:53.887194 29966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:53.887979 29966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:53.889230 29966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:54:53.889297 29966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:53.889369 29966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:54:53.889407 29966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:53.920606 29966 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:46439
I20260806 04:54:53.920650 30096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:46439 every 8 connection(s)
I20260806 04:54:53.921881 29966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:54:53.926856 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 29966
I20260806 04:54:53.927054 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:54:53.927608 13906 external_mini_cluster.cc:1428] Setting key 366deac1d4ec97ceeb4df33b8f6cc956
Aug 06 04:54:53 dist-test-slave-1x4g krb5kdc[29722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992093, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:53.942365 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:53.924364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:42413 (local address 127.13.148.190:41047)
0806 04:54:53.924561 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:53.924567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:53.925654 (+ 1087us) server_negotiation.cc:407] Connection header received
0806 04:54:53.927677 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:53.927684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:53.927780 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:53.927862 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:53.929642 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:53.930247 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:53.932522 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:53.933001 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:53.933095 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:53.937069 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:53.937074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:53.937078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:53.937107 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:53.939641 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:53.940300 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:53.940304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:53.940306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:53.940424 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:53.940898 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:53.940902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:53.940905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:53.941131 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:53.941270 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:53.941844 (+ 574us) server_negotiation.cc:299] Negotiation successful
0806 04:54:53.942046 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 04:54:53.943325 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:53.924898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:42413)
0806 04:54:53.925514 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:53.925549 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:53.927384 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:53.928125 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:53.928136 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:53.928853 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:53.929469 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:53.929488 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:53.930388 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:53.930395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:53.932386 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:53.932401 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:53.933216 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:53.933226 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:53.933560 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:53.934278 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:53.934311 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:53.936854 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:53.939772 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:53.939780 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:53.939794 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:53.940175 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:53.940545 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:53.940549 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:53.940552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:53.940740 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:53.941247 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:53.941254 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:53.941537 (+ 283us) client_negotiation.cc:769] Sending connection context
0806 04:54:53.941761 (+ 224us) client_negotiation.cc:240] Negotiation successful
0806 04:54:53.942219 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":227,"threads_started":1}
I20260806 04:54:53.944679 30097 heartbeater.cc:347] Connected to a master server at 127.13.148.190:41047
I20260806 04:54:53.945019 30097 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:53.945755 30097 heartbeater.cc:511] Master 127.13.148.190:41047 requested a full tablet report, sending...
I20260806 04:54:53.947217 29767 ts_manager.cc:194] Registered new tserver with Master: e4fda8f8b55d45c9bef0073d6278d7f3 (127.13.148.130:46439)
I20260806 04:54:53.948354 29767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:42413
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
I20260806 04:54:53.983464 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41047
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:54:54.281826 30102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:54:54.282065 30102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:54:54.282148 30102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:54:54.288352 30102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:54:54.288450 30102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:54:54.288563 30102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:54:54.296957 30102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:54:54.298074 30102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:54:54.299193 30102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:54:54.311131 30107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:54.311087 30108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:54:54.311218 30110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:54:54.311830 30102 server_base.cc:1034] running on GCE node
I20260806 04:54:54.312394 30102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:54:54.313396 30102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:54:54.314621 30102 hybrid_clock.cc:648] HybridClock initialized: now 1785992094314565 us; error 46 us; skew 500 ppm
I20260806 04:54:54.314929 30102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:54:54 dist-test-slave-1x4g krb5kdc[29722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992094, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:54:54.318027 30102 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:54:54.319274 30102 webserver.cc:460] Webserver started at http://127.13.148.131:44239/ using document root <none> and password file <none>
I20260806 04:54:54.319900 30102 fs_manager.cc:359] Metadata directory not provided
I20260806 04:54:54.320041 30102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:54:54.320333 30102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:54:54.323112 30102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "4475889dd7e04809864df3a223749914"
format_stamp: "Formatted at 2026-08-06 04:54:54 on dist-test-slave-1x4g"
server_key: "495df165ed8cda7339a0281786981702"
server_key_iv: "e44c63f39f500e85333767016b74d0db"
server_key_version: "encryptionkey@0"
I20260806 04:54:54.323936 30102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "4475889dd7e04809864df3a223749914"
format_stamp: "Formatted at 2026-08-06 04:54:54 on dist-test-slave-1x4g"
server_key: "495df165ed8cda7339a0281786981702"
server_key_iv: "e44c63f39f500e85333767016b74d0db"
server_key_version: "encryptionkey@0"
I20260806 04:54:54.328917 30102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260806 04:54:54.332188 30117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:54.333501 30102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:54:54.333683 30102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "4475889dd7e04809864df3a223749914"
format_stamp: "Formatted at 2026-08-06 04:54:54 on dist-test-slave-1x4g"
server_key: "495df165ed8cda7339a0281786981702"
server_key_iv: "e44c63f39f500e85333767016b74d0db"
server_key_version: "encryptionkey@0"
I20260806 04:54:54.333905 30102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:54:54.347790 30102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:54:54.351188 30102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:54:54.351500 30102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:54:54.352327 30102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:54:54.353681 30102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:54:54.353767 30102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:54.353976 30102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:54:54.354043 30102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:54:54.386349 30102 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:44299
I20260806 04:54:54.386379 30233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:44299 every 8 connection(s)
I20260806 04:54:54.387542 30102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:54:54.394461 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 30102
I20260806 04:54:54.394665 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:54:54.395115 13906 external_mini_cluster.cc:1428] Setting key 6377db4fc7a6f059138a023dacb23d28
Aug 06 04:54:54 dist-test-slave-1x4g krb5kdc[29722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992094, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:54:54.406358 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:54.389812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:55891 (local address 127.13.148.190:41047)
0806 04:54:54.390036 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:54.390055 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:54.391104 (+ 1049us) server_negotiation.cc:407] Connection header received
0806 04:54:54.392753 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:54.392759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:54.392890 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:54.393010 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:54.394642 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.395453 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.397167 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.397617 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.397749 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:54.401122 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:54.401127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:54.401130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:54.401159 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:54.403588 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:54.404264 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:54.404270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:54.404273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:54.404386 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:54.404797 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:54.404803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:54.404806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:54.405065 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:54.405207 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:54.405829 (+ 622us) server_negotiation.cc:299] Negotiation successful
0806 04:54:54.406045 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 04:54:54.407222 30236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:54.390303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:55891)
0806 04:54:54.390946 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:54.390999 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:54.392476 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:54.393236 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:54.393244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:54.393742 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:54:54.394419 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:54.394448 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.395605 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.395608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:54.397016 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:54.397029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.397842 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.397850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:54.398106 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:54.398720 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:54:54.398739 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:54:54.400903 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:54:54.403719 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:54.403727 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:54.403742 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:54.404151 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:54.404493 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:54:54.404498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:54:54.404501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:54:54.404676 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:54:54.405191 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:54:54.405198 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:54:54.405493 (+ 295us) client_negotiation.cc:769] Sending connection context
0806 04:54:54.405750 (+ 257us) client_negotiation.cc:240] Negotiation successful
0806 04:54:54.406203 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":251,"threads_started":1}
I20260806 04:54:54.408838 30234 heartbeater.cc:347] Connected to a master server at 127.13.148.190:41047
I20260806 04:54:54.409150 30234 heartbeater.cc:464] Registering TS with master...
I20260806 04:54:54.409965 30234 heartbeater.cc:511] Master 127.13.148.190:41047 requested a full tablet report, sending...
I20260806 04:54:54.411405 29767 ts_manager.cc:194] Registered new tserver with Master: 4475889dd7e04809864df3a223749914 (127.13.148.131:44299)
I20260806 04:54:54.412153 29767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:55891
I20260806 04:54:54.422394 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:54:54.438083 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:54.426551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.13.148.190:41047)
0806 04:54:54.426741 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:54.426748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:54.427090 (+ 342us) server_negotiation.cc:407] Connection header received
0806 04:54:54.427199 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:54.427202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:54.427252 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:54.427319 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:54.428508 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.429179 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.430304 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.430661 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.430738 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:54.432909 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:54.432914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:54.432917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:54.432952 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:54.435420 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:54.436119 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:54.436122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:54.436124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:54.436241 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:54.436553 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:54.436556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:54.436558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:54.437059 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:54.437235 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:54.437505 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 04:54:54.437748 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:54:54.441605 29767 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 04:54:54.446615 29960 heartbeater.cc:503] Master 127.13.148.190:41047 was elected leader, sending a full tablet report...
W20260806 04:54:54.447818 29767 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:54:54.472828 30247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:54.458700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:46439 (local address 127.0.0.1:53624)
0806 04:54:54.459374 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:54.459422 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:54.459603 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:54.464881 (+ 5278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:54.464891 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:54.464933 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:54.465637 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:54.465669 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.467198 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.467203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:54.469547 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:54.469562 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.471572 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.471586 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:54.471903 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:54.472017 (+ 114us) client_negotiation.cc:769] Sending connection context
0806 04:54:54.472159 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 04:54:54.472364 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":118,"spinlock_wait_cycles":10496,"thread_start_us":904,"threads_started":1}
I20260806 04:54:54.473333 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:54.459131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.13.148.131:44299)
0806 04:54:54.464583 (+ 5452us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:54.464590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:54.464616 (+ 26us) server_negotiation.cc:407] Connection header received
0806 04:54:54.464700 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:54.464706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:54.464912 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:54.465082 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:54.465755 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.467238 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.470697 (+ 3459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.471992 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.472164 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:54.472332 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:54.472808 (+ 476us) server_negotiation.cc:299] Negotiation successful
0806 04:54:54.472962 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5304,"thread_start_us":5362,"threads_started":1}
I20260806 04:54:54.473491 30250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:54.460030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36149 (local address 127.0.0.1:43718)
0806 04:54:54.460571 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:54.460595 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:54.460714 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:54.466234 (+ 5520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:54.466239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:54.466263 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:54.466641 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:54.466658 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.468235 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.468242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:54.470182 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:54.470207 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.471598 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.471604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:54.471940 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:54.472911 (+ 971us) client_negotiation.cc:769] Sending connection context
0806 04:54:54.473016 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 04:54:54.473152 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":995,"thread_start_us":257,"threads_started":1}
I20260806 04:54:54.473958 30246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:54.458636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:44299 (local address 127.0.0.1:58900)
0806 04:54:54.459376 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:54.459425 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:54.459603 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:54.465077 (+ 5474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:54.465081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:54.465102 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:54.465552 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:54.465565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.467407 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.467411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:54.470516 (+ 3105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:54.470530 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.472473 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.472481 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:54.472603 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:54.472623 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 04:54:54.472743 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 04:54:54.472883 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":50,"spinlock_wait_cycles":5760,"thread_start_us":771,"threads_started":1}
I20260806 04:54:54.474150 30251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:54.460409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.13.148.129:36149)
0806 04:54:54.465737 (+ 5328us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:54.465746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:54.465768 (+ 22us) server_negotiation.cc:407] Connection header received
0806 04:54:54.465845 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:54.465851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:54.466051 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:54.466218 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:54.466797 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.468131 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.470344 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.471287 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.472072 (+ 785us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:54.472269 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:54.473716 (+ 1447us) server_negotiation.cc:299] Negotiation successful
0806 04:54:54.473871 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5154,"thread_start_us":5010,"threads_started":1}
I20260806 04:54:54.474583 30248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:54.459234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53624 (local address 127.13.148.130:46439)
0806 04:54:54.464262 (+ 5028us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:54.464272 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:54.464301 (+ 29us) server_negotiation.cc:407] Connection header received
0806 04:54:54.464395 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:54.464403 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:54.464642 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:54.464836 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:54.465822 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.467054 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.469728 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.471112 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.471241 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:54.471416 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:54.474172 (+ 2756us) server_negotiation.cc:299] Negotiation successful
0806 04:54:54.474333 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4886,"thread_start_us":4787,"threads_started":1}
I20260806 04:54:54.477978 29893 tablet_service.cc:1467] Processing CreateTablet for tablet a2fecab380904f27ad7d6343596d33c5 (DEFAULT_TABLE table=test-table [id=9d1bcc55315e45de9c84e9d329c9256e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:54:54.478016 30032 tablet_service.cc:1467] Processing CreateTablet for tablet a2fecab380904f27ad7d6343596d33c5 (DEFAULT_TABLE table=test-table [id=9d1bcc55315e45de9c84e9d329c9256e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:54:54.477906 30167 tablet_service.cc:1467] Processing CreateTablet for tablet a2fecab380904f27ad7d6343596d33c5 (DEFAULT_TABLE table=test-table [id=9d1bcc55315e45de9c84e9d329c9256e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:54:54.479140 29893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2fecab380904f27ad7d6343596d33c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:54.479228 30167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2fecab380904f27ad7d6343596d33c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:54.479229 30032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2fecab380904f27ad7d6343596d33c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:54:54.488726 30252 tablet_bootstrap.cc:492] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063: Bootstrap starting.
I20260806 04:54:54.492125 30252 tablet_bootstrap.cc:654] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:54.492573 30254 tablet_bootstrap.cc:492] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3: Bootstrap starting.
I20260806 04:54:54.493563 30252 log.cc:824] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:54.494918 30253 tablet_bootstrap.cc:492] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914: Bootstrap starting.
I20260806 04:54:54.495877 30254 tablet_bootstrap.cc:654] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:54.497040 30254 log.cc:824] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:54.497099 30252 tablet_bootstrap.cc:492] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063: No bootstrap required, opened a new log
I20260806 04:54:54.497344 30252 ts_tablet_manager.cc:1397] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 04:54:54.499898 30254 tablet_bootstrap.cc:492] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3: No bootstrap required, opened a new log
I20260806 04:54:54.499945 30253 tablet_bootstrap.cc:654] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914: Neither blocks nor log segments found. Creating new log.
I20260806 04:54:54.500152 30254 ts_tablet_manager.cc:1397] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260806 04:54:54.501174 30253 log.cc:824] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914: Log is configured to *not* fsync() on all Append() calls
I20260806 04:54:54.504092 30253 tablet_bootstrap.cc:492] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914: No bootstrap required, opened a new log
I20260806 04:54:54.504333 30253 ts_tablet_manager.cc:1397] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260806 04:54:54.505056 30252 raft_consensus.cc:348] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } }
I20260806 04:54:54.505359 30252 raft_consensus.cc:374] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:54.505470 30252 raft_consensus.cc:729] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d1b567085d45a080c2dda2c9bdb063, State: Initialized, Role: FOLLOWER
I20260806 04:54:54.505999 30252 consensus_queue.cc:260] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } }
I20260806 04:54:54.507468 30252 ts_tablet_manager.cc:1428] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260806 04:54:54.510726 30254 raft_consensus.cc:348] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } }
I20260806 04:54:54.511125 30254 raft_consensus.cc:374] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:54.511221 30254 raft_consensus.cc:729] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4fda8f8b55d45c9bef0073d6278d7f3, State: Initialized, Role: FOLLOWER
I20260806 04:54:54.511749 30254 consensus_queue.cc:260] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } }
I20260806 04:54:54.512673 30097 heartbeater.cc:503] Master 127.13.148.190:41047 was elected leader, sending a full tablet report...
I20260806 04:54:54.513525 30254 ts_tablet_manager.cc:1428] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260806 04:54:54.514047 30253 raft_consensus.cc:348] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } }
I20260806 04:54:54.514453 30253 raft_consensus.cc:374] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:54:54.514562 30253 raft_consensus.cc:729] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4475889dd7e04809864df3a223749914, State: Initialized, Role: FOLLOWER
I20260806 04:54:54.515120 30253 consensus_queue.cc:260] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } }
I20260806 04:54:54.516202 30234 heartbeater.cc:503] Master 127.13.148.190:41047 was elected leader, sending a full tablet report...
I20260806 04:54:54.516737 30253 ts_tablet_manager.cc:1428] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260806 04:54:54.540167 30258 raft_consensus.cc:482] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:54:54.540385 30258 raft_consensus.cc:504] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } }
I20260806 04:54:54.541827 30258 leader_election.cc:288] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4475889dd7e04809864df3a223749914 (127.13.148.131:44299), e4fda8f8b55d45c9bef0073d6278d7f3 (127.13.148.130:46439)
I20260806 04:54:54.548112 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:54.542135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:59011 (local address 127.13.148.131:44299)
0806 04:54:54.542339 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:54.542347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:54.543136 (+ 789us) server_negotiation.cc:407] Connection header received
0806 04:54:54.543361 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:54.543366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:54.543446 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:54.543537 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:54.544175 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.544884 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.546599 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.547281 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.547413 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:54.547498 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:54.547779 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 04:54:54.547883 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:54:54.548148 30248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:54.542293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:44031 (local address 127.13.148.130:46439)
0806 04:54:54.542487 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:54.542494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:54.543294 (+ 800us) server_negotiation.cc:407] Connection header received
0806 04:54:54.543473 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:54.543477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:54.543559 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:54.543646 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:54:54.544257 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.544966 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.546658 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.547300 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.547452 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:54.547516 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:54.547813 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 04:54:54.547921 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:54:54.548327 30261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:54.542024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:44299 (local address 127.13.148.129:59011)
0806 04:54:54.543026 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:54.543041 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:54.543203 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:54.543575 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:54.543582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:54.543613 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:54.544046 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:54.544057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.544996 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.545002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:54.546457 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:54.546472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.547485 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.547491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:54.547622 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:54.547664 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 04:54:54.547730 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 04:54:54.547852 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":60,"thread_start_us":631,"threads_started":1}
I20260806 04:54:54.548375 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:54.542293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:46439 (local address 127.13.148.129:44031)
0806 04:54:54.543136 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0806 04:54:54.543194 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 04:54:54.543348 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:54:54.543696 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:54:54.543701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:54:54.543743 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:54:54.544079 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:54.544086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.545095 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.545098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:54.546495 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:54:54.546507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.547531 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.547536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:54:54.547662 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:54.547681 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:54:54.547777 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 04:54:54.547883 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":38,"thread_start_us":521,"threads_started":1}
I20260806 04:54:54.549191 30052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fecab380904f27ad7d6343596d33c5" candidate_uuid: "36d1b567085d45a080c2dda2c9bdb063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" is_pre_election: true
I20260806 04:54:54.549208 30188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fecab380904f27ad7d6343596d33c5" candidate_uuid: "36d1b567085d45a080c2dda2c9bdb063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4475889dd7e04809864df3a223749914" is_pre_election: true
I20260806 04:54:54.549597 30052 raft_consensus.cc:2454] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36d1b567085d45a080c2dda2c9bdb063 in term 0.
I20260806 04:54:54.549611 30188 raft_consensus.cc:2454] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36d1b567085d45a080c2dda2c9bdb063 in term 0.
I20260806 04:54:54.550143 29847 leader_election.cc:302] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d1b567085d45a080c2dda2c9bdb063, e4fda8f8b55d45c9bef0073d6278d7f3; no voters:
I20260806 04:54:54.550484 30258 raft_consensus.cc:2781] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:54:54.550622 30258 raft_consensus.cc:482] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:54:54.550717 30258 raft_consensus.cc:3037] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:54.552520 30258 raft_consensus.cc:504] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } }
I20260806 04:54:54.553164 30258 leader_election.cc:288] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [CANDIDATE]: Term 1 election: Requested vote from peers 4475889dd7e04809864df3a223749914 (127.13.148.131:44299), e4fda8f8b55d45c9bef0073d6278d7f3 (127.13.148.130:46439)
I20260806 04:54:54.553542 30188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fecab380904f27ad7d6343596d33c5" candidate_uuid: "36d1b567085d45a080c2dda2c9bdb063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4475889dd7e04809864df3a223749914"
I20260806 04:54:54.553615 30052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fecab380904f27ad7d6343596d33c5" candidate_uuid: "36d1b567085d45a080c2dda2c9bdb063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3"
I20260806 04:54:54.553725 30188 raft_consensus.cc:3037] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:54.553835 30052 raft_consensus.cc:3037] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:54:54.555918 30188 raft_consensus.cc:2454] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d1b567085d45a080c2dda2c9bdb063 in term 1.
I20260806 04:54:54.556074 30052 raft_consensus.cc:2454] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d1b567085d45a080c2dda2c9bdb063 in term 1.
I20260806 04:54:54.556291 29849 leader_election.cc:302] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d1b567085d45a080c2dda2c9bdb063, 4475889dd7e04809864df3a223749914; no voters:
I20260806 04:54:54.556572 30258 raft_consensus.cc:2781] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:54:54.557106 30258 raft_consensus.cc:686] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [term 1 LEADER]: Becoming Leader. State: Replica: 36d1b567085d45a080c2dda2c9bdb063, State: Running, Role: LEADER
I20260806 04:54:54.557605 30258 consensus_queue.cc:237] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } }
I20260806 04:54:54.561532 29767 catalog_manager.cc:5495] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36d1b567085d45a080c2dda2c9bdb063 (127.13.148.129). New cstate: current_term: 1 leader_uuid: "36d1b567085d45a080c2dda2c9bdb063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } health_report { overall_health: HEALTHY } } }
I20260806 04:54:54.591454 30251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:54.586523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.13.148.129:36149)
0806 04:54:54.586706 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:54.586712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:54.586831 (+ 119us) server_negotiation.cc:407] Connection header received
0806 04:54:54.587045 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:54.587048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:54.587113 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:54.587244 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:54:54.587612 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.588221 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.589514 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.589871 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.589988 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:54.590464 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:54:54.590934 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:54:54.591080 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:54.591152 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 04:54:54.591221 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:54:54.598896 30052 raft_consensus.cc:1261] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 1 FOLLOWER]: Refusing update from remote peer 36d1b567085d45a080c2dda2c9bdb063: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:54:54.598897 30188 raft_consensus.cc:1261] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914 [term 1 FOLLOWER]: Refusing update from remote peer 36d1b567085d45a080c2dda2c9bdb063: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:54:54.599736 30263 consensus_queue.cc:1035] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:54:54.600028 30258 consensus_queue.cc:1035] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:54:54.624629 29766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:54:54.624986 29766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:54:54.628156 29766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Sending DeleteTablet for 3 replicas of tablet a2fecab380904f27ad7d6343596d33c5
I20260806 04:54:54.629232 30167 tablet_service.cc:1514] Processing DeleteTablet for tablet a2fecab380904f27ad7d6343596d33c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:54:54 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:58900
I20260806 04:54:54.629232 30032 tablet_service.cc:1514] Processing DeleteTablet for tablet a2fecab380904f27ad7d6343596d33c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:54:54 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:53624
I20260806 04:54:54.629294 29893 tablet_service.cc:1514] Processing DeleteTablet for tablet a2fecab380904f27ad7d6343596d33c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:54:54 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:43718
I20260806 04:54:54.629997 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 29829
I20260806 04:54:54.630537 30276 tablet_replica.cc:330] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3: stopping tablet replica
I20260806 04:54:54.630743 30277 tablet_replica.cc:330] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914: stopping tablet replica
I20260806 04:54:54.631695 30276 raft_consensus.cc:2229] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:54:54.632305 30277 raft_consensus.cc:2229] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:54:54.633213 30277 raft_consensus.cc:2258] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:54:54.633488 30276 raft_consensus.cc:2258] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:54:54.636134 30277 ts_tablet_manager.cc:1905] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:54:54.637616 30276 ts_tablet_manager.cc:1905] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:54:54.645051 30278 tablet_replica.cc:330] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063: stopping tablet replica
I20260806 04:54:54.645413 30278 raft_consensus.cc:2229] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:54:54.645987 30278 raft_consensus.cc:2258] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:54:54.647472 30278 ts_tablet_manager.cc:1905] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:54:54.650043 30276 ts_tablet_manager.cc:1918] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:54:54.652500 30277 ts_tablet_manager.cc:1918] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:54:54.652722 30276 log.cc:1196] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/a2fecab380904f27ad7d6343596d33c5
Aug 06 04:54:54 dist-test-slave-1x4g krb5kdc[29722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:54:54.653400 30276 ts_tablet_manager.cc:1939] T a2fecab380904f27ad7d6343596d33c5 P e4fda8f8b55d45c9bef0073d6278d7f3: Deleting consensus metadata
I20260806 04:54:54.655193 29752 catalog_manager.cc:4841] TS e4fda8f8b55d45c9bef0073d6278d7f3 (127.13.148.130:46439): tablet a2fecab380904f27ad7d6343596d33c5 (table test-table [id=9d1bcc55315e45de9c84e9d329c9256e]) successfully deleted
I20260806 04:54:54.655648 30277 log.cc:1196] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/a2fecab380904f27ad7d6343596d33c5
I20260806 04:54:54.656348 30277 ts_tablet_manager.cc:1939] T a2fecab380904f27ad7d6343596d33c5 P 4475889dd7e04809864df3a223749914: Deleting consensus metadata
I20260806 04:54:54.658109 29754 catalog_manager.cc:4841] TS 4475889dd7e04809864df3a223749914 (127.13.148.131:44299): tablet a2fecab380904f27ad7d6343596d33c5 (table test-table [id=9d1bcc55315e45de9c84e9d329c9256e]) successfully deleted
I20260806 04:54:54.658898 30278 ts_tablet_manager.cc:1918] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:54:54.659087 30278 log.cc:1196] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/a2fecab380904f27ad7d6343596d33c5
I20260806 04:54:54.659471 30278 ts_tablet_manager.cc:1939] T a2fecab380904f27ad7d6343596d33c5 P 36d1b567085d45a080c2dda2c9bdb063: Deleting consensus metadata
I20260806 04:54:54.660655 29754 catalog_manager.cc:4841] TS 36d1b567085d45a080c2dda2c9bdb063 (127.13.148.129:36149): tablet a2fecab380904f27ad7d6343596d33c5 (table test-table [id=9d1bcc55315e45de9c84e9d329c9256e]) successfully deleted
I20260806 04:54:54.662238 30251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:54.631073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.13.148.129:36149)
0806 04:54:54.631289 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:54.631298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:54.641792 (+ 10494us) server_negotiation.cc:407] Connection header received
0806 04:54:54.642161 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:54.642166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:54.642233 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:54.642301 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:54.644352 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.645662 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.647711 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.648362 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.648461 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:54.654139 (+ 5678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:54.654151 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:54.654166 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:54.654205 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:54.658818 (+ 4613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:54.659777 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:54.659787 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:54.659794 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:54.659961 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:54.660464 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:54.660469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:54.660473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:54.660941 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:54.661243 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:54.661823 (+ 580us) server_negotiation.cc:299] Negotiation successful
0806 04:54:54.661997 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 04:54:54.663017 29955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:54.751102 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 29829
I20260806 04:54:54.770342 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 29966
Aug 06 04:54:54 dist-test-slave-1x4g krb5kdc[29722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:54:54.784578 30248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:54.771083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.13.148.130:46439)
0806 04:54:54.771254 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:54.771261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:54.771278 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:54:54.771463 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:54.771467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:54.771524 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:54.771628 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:54.772820 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.773642 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.774891 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.775263 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.775397 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:54.779346 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:54.779357 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:54.779371 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:54.779399 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:54.781920 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:54.782577 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:54.782585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:54.782588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:54.782711 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:54.783109 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:54.783115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:54.783118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:54.783487 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:54.783799 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:54.784101 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 04:54:54.784288 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:54:54.785195 30092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:54.867882 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 29966
I20260806 04:54:54.885993 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 30102
Aug 06 04:54:54 dist-test-slave-1x4g krb5kdc[29722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
I20260806 04:54:54.900414 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:54:54.886746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58912 (local address 127.13.148.131:44299)
0806 04:54:54.886887 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 04:54:54.886891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:54:54.886907 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:54:54.887000 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:54:54.887004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:54:54.887059 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:54:54.887131 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:54:54.888313 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.889268 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.890432 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:54:54.890765 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:54:54.890903 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:54:54.895094 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:54:54.895105 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:54:54.895120 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:54:54.895154 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:54:54.897797 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:54.898475 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:54.898481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:54.898483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:54.898595 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:54:54.898954 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:54:54.898962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:54:54.898965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:54:54.899297 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:54:54.899595 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 04:54:54.899906 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 04:54:54.900107 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 04:54:54.901077 30229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:54.980633 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 30102
I20260806 04:54:54.998183 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 29735
I20260806 04:54:54.999228 29797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:54:55.102080 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 29735
I20260806 04:55:15.125310 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41047
--webserver_interface=127.13.148.190
--webserver_port=40965
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:41047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:55:15.443205 30284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:55:15.443452 30284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:55:15.443528 30284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:55:15.449713 30284 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 04:55:15.449793 30284 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 04:55:15.449829 30284 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 04:55:15.449883 30284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:55:15.449932 30284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:55:15.449994 30284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:55:15.450035 30284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:55:15.450079 30284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:55:15.458878 30284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:41047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=40965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:55:15.460000 30284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:55:15.461221 30284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:55:15.482867 30293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:15.483022 30290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:15.482905 30291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:55:15.483512 30284 server_base.cc:1034] running on GCE node
I20260806 04:55:15.484344 30284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:55:15.485852 30284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:55:15.487075 30284 hybrid_clock.cc:648] HybridClock initialized: now 1785992115487021 us; error 24 us; skew 500 ppm
I20260806 04:55:15.487386 30284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:55:15 dist-test-slave-1x4g krb5kdc[29722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992115, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:55:15.490764 30284 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:55:15.492304 30284 webserver.cc:460] Webserver started at http://127.13.148.190:40965/ using document root <none> and password file <none>
I20260806 04:55:15.493081 30284 fs_manager.cc:359] Metadata directory not provided
I20260806 04:55:15.493207 30284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:55:15.498792 30284 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260806 04:55:15.501585 30301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:15.502862 30284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:55:15.503033 30284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "a79f52b5e4274220b876f5422483397c"
format_stamp: "Formatted at 2026-08-06 04:54:52 on dist-test-slave-1x4g"
server_key: "6e2608cab18140526aa8ee3c77fdd452"
server_key_iv: "26555c209b659eeaacfced78e4055c91"
server_key_version: "encryptionkey@0"
I20260806 04:55:15.503970 30284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:55:15.526512 30284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:55:15.530246 30284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:55:15.530534 30284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:55:15.548314 30284 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:41047
I20260806 04:55:15.548336 30352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:41047 every 8 connection(s)
I20260806 04:55:15.549846 30284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:55:15.550788 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 30284
I20260806 04:55:15.551748 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:36149
--local_ip_for_outbound_sockets=127.13.148.129
--tserver_master_addrs=127.13.148.190:41047
--webserver_port=40687
--webserver_interface=127.13.148.129
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:55:15.559836 30353 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:55:15.564062 30353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Bootstrap starting.
I20260806 04:55:15.582864 30353 log.cc:824] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:55:15.593832 30353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:55:15.594388 30353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Bootstrap complete.
I20260806 04:55:15.602782 30353 raft_consensus.cc:348] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } }
I20260806 04:55:15.603684 30353 raft_consensus.cc:729] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a79f52b5e4274220b876f5422483397c, State: Initialized, Role: FOLLOWER
I20260806 04:55:15.604236 30353 consensus_queue.cc:260] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } }
I20260806 04:55:15.604518 30353 raft_consensus.cc:388] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:55:15.604650 30353 raft_consensus.cc:482] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:55:15.604887 30353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:55:15.608485 30353 raft_consensus.cc:504] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } }
I20260806 04:55:15.609038 30353 leader_election.cc:302] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a79f52b5e4274220b876f5422483397c; no voters:
I20260806 04:55:15.609669 30353 leader_election.cc:288] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 04:55:15.609786 30357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:55:15.611935 30353 sys_catalog.cc:564] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:55:15.613590 30357 raft_consensus.cc:686] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 2 LEADER]: Becoming Leader. State: Replica: a79f52b5e4274220b876f5422483397c, State: Running, Role: LEADER
I20260806 04:55:15.614161 30357 consensus_queue.cc:237] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } }
I20260806 04:55:15.618577 30358 sys_catalog.cc:455] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a79f52b5e4274220b876f5422483397c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } } }
I20260806 04:55:15.618988 30358 sys_catalog.cc:458] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [sys.catalog]: This master's current role is: LEADER
I20260806 04:55:15.620317 30359 sys_catalog.cc:455] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a79f52b5e4274220b876f5422483397c. Latest consensus state: current_term: 2 leader_uuid: "a79f52b5e4274220b876f5422483397c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } } }
I20260806 04:55:15.620631 30359 sys_catalog.cc:458] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [sys.catalog]: This master's current role is: LEADER
I20260806 04:55:15.624426 30370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:55:15.629856 30370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:55:15.630750 30370 catalog_manager.cc:1259] Loaded cluster ID: ce1261c06295411fbb437b0bce6b5f4d
I20260806 04:55:15.630854 30370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:55:15.633764 30370 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:55:15.634812 30370 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Loaded TSK: 0
I20260806 04:55:15.652393 30370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Generated new TSK 1
I20260806 04:55:15.652688 30370 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 04:55:15.896556 30355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:55:15.896950 30355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:55:15.897120 30355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:55:15.903667 30355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:55:15.903800 30355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:55:15.903973 30355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:55:15.913405 30355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:36149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=40687
--tserver_master_addrs=127.13.148.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:55:15.914661 30355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:55:15.916055 30355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:55:15.930544 30383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:15.930635 30381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:15.930935 30380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:55:15.932168 30355 server_base.cc:1034] running on GCE node
I20260806 04:55:15.932730 30355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:55:15.933701 30355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:55:15.934922 30355 hybrid_clock.cc:648] HybridClock initialized: now 1785992115934864 us; error 50 us; skew 500 ppm
I20260806 04:55:15.935235 30355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:55:15 dist-test-slave-1x4g krb5kdc[29722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992115, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:55:15.938647 30355 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:55:15.940199 30355 webserver.cc:460] Webserver started at http://127.13.148.129:40687/ using document root <none> and password file <none>
I20260806 04:55:15.940892 30355 fs_manager.cc:359] Metadata directory not provided
I20260806 04:55:15.941039 30355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:55:15.945971 30355 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:55:15.948737 30390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:15.950340 30355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:55:15.950513 30355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "36d1b567085d45a080c2dda2c9bdb063"
format_stamp: "Formatted at 2026-08-06 04:54:53 on dist-test-slave-1x4g"
server_key: "8f32e550d2cbbd33a9648bfcb0825a8a"
server_key_iv: "0f95f0c46d3b4b86102bc3c2b2d216ff"
server_key_version: "encryptionkey@0"
I20260806 04:55:15.951470 30355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:55:15.984254 30355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:55:15.988073 30355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:55:15.988340 30355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:55:15.989344 30355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:55:15.990569 30355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:55:15.990685 30355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:15.990869 30355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:55:15.990954 30355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:16.024459 30355 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:36149
I20260806 04:55:16.024608 30503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:36149 every 8 connection(s)
I20260806 04:55:16.025972 30355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:55:16.029853 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 30355
I20260806 04:55:16.030467 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:46439
--local_ip_for_outbound_sockets=127.13.148.130
--tserver_master_addrs=127.13.148.190:41047
--webserver_port=38069
--webserver_interface=127.13.148.130
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:55:16 dist-test-slave-1x4g krb5kdc[29722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992115, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:55:16.049216 30506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:16.028611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:48585 (local address 127.13.148.190:41047)
0806 04:55:16.029191 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:16.029202 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:16.029825 (+ 623us) server_negotiation.cc:407] Connection header received
0806 04:55:16.031892 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:16.031912 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:16.032308 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:16.032714 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:16.034188 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.036112 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.037680 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.038813 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.038998 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:16.042462 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:16.042469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:16.042491 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:16.042533 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:16.045254 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:16.046155 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:16.046161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:16.046165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:16.046307 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:16.046766 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:16.046769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:16.046771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:16.047131 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:16.047266 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:16.048116 (+ 850us) server_negotiation.cc:299] Negotiation successful
0806 04:55:16.048369 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":277,"threads_started":1}
I20260806 04:55:16.049649 30507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:16.028756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:48585)
0806 04:55:16.029628 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:16.029671 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 04:55:16.031326 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:55:16.032882 (+ 1556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:55:16.032889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:55:16.033343 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:55:16.034000 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:16.034019 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.036264 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.036267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:16.037538 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:16.037551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.039023 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.039027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:16.039204 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:16.039896 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:55:16.039916 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:55:16.042218 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:55:16.045392 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:55:16.045412 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:55:16.045428 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:55:16.046039 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:55:16.046434 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:55:16.046440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:55:16.046446 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:55:16.046663 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:55:16.047275 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:55:16.047284 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:55:16.047669 (+ 385us) client_negotiation.cc:769] Sending connection context
0806 04:55:16.047950 (+ 281us) client_negotiation.cc:240] Negotiation successful
0806 04:55:16.048448 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":329,"threads_started":1}
I20260806 04:55:16.052444 30504 heartbeater.cc:347] Connected to a master server at 127.13.148.190:41047
I20260806 04:55:16.052748 30504 heartbeater.cc:464] Registering TS with master...
I20260806 04:55:16.053483 30504 heartbeater.cc:511] Master 127.13.148.190:41047 requested a full tablet report, sending...
I20260806 04:55:16.056051 30318 ts_manager.cc:194] Registered new tserver with Master: 36d1b567085d45a080c2dda2c9bdb063 (127.13.148.129:36149)
I20260806 04:55:16.058422 30318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:48585
W20260806 04:55:16.336858 30508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:55:16.337103 30508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:55:16.337188 30508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:55:16.343434 30508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:55:16.343544 30508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:55:16.343650 30508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:55:16.351867 30508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:46439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=38069
--tserver_master_addrs=127.13.148.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:55:16.352907 30508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:55:16.354086 30508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:55:16.366086 30513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:16.366120 30514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:16.366050 30516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:55:16.367056 30508 server_base.cc:1034] running on GCE node
I20260806 04:55:16.367550 30508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:55:16.368306 30508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:55:16.369478 30508 hybrid_clock.cc:648] HybridClock initialized: now 1785992116369438 us; error 31 us; skew 500 ppm
I20260806 04:55:16.369704 30508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:55:16 dist-test-slave-1x4g krb5kdc[29722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992116, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:55:16.372442 30508 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:55:16.373673 30508 webserver.cc:460] Webserver started at http://127.13.148.130:38069/ using document root <none> and password file <none>
I20260806 04:55:16.374285 30508 fs_manager.cc:359] Metadata directory not provided
I20260806 04:55:16.374465 30508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:55:16.379194 30508 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260806 04:55:16.381464 30523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:16.382584 30508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:55:16.382730 30508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "e4fda8f8b55d45c9bef0073d6278d7f3"
format_stamp: "Formatted at 2026-08-06 04:54:53 on dist-test-slave-1x4g"
server_key: "1c47c0ebfec6bde4c167d911a546e37c"
server_key_iv: "8c571feac16eda73ab981e7544f9fc77"
server_key_version: "encryptionkey@0"
I20260806 04:55:16.383535 30508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:55:16.396957 30508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:55:16.400225 30508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:55:16.400455 30508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:55:16.401273 30508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:55:16.402352 30508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:55:16.402449 30508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:16.402565 30508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:55:16.402643 30508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:16.433197 30508 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:46439
I20260806 04:55:16.433264 30637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:46439 every 8 connection(s)
I20260806 04:55:16.434458 30508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:55:16.438211 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 30508
I20260806 04:55:16.438865 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:44299
--local_ip_for_outbound_sockets=127.13.148.131
--tserver_master_addrs=127.13.148.190:41047
--webserver_port=44239
--webserver_interface=127.13.148.131
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:55:16 dist-test-slave-1x4g krb5kdc[29722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992116, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:55:16.453828 30506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:16.436738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:47031 (local address 127.13.148.190:41047)
0806 04:55:16.436968 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:16.436974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:16.438128 (+ 1154us) server_negotiation.cc:407] Connection header received
0806 04:55:16.440063 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:16.440068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:16.440146 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:16.440242 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:16.441612 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.442220 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.443559 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.443942 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.444066 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:16.447751 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:16.447758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:16.447765 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:16.447808 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:16.450979 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:16.451588 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:16.451595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:16.451600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:16.451733 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:16.452095 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:16.452102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:16.452107 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:16.452444 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:16.452580 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:16.453171 (+ 591us) server_negotiation.cc:299] Negotiation successful
0806 04:55:16.453410 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 04:55:16.454391 30640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:16.437276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:47031)
0806 04:55:16.437993 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:16.438026 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 04:55:16.439819 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:55:16.440458 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:55:16.440465 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:55:16.440974 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:55:16.441446 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:16.441471 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.442366 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.442374 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:16.443417 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:16.443428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.444141 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.444148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:16.444386 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:16.445080 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:55:16.445113 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:55:16.447477 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:55:16.451115 (+ 3638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:55:16.451121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:55:16.451130 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:55:16.451466 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:55:16.451827 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:55:16.451830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:55:16.451832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:55:16.451978 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:55:16.452550 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:55:16.452555 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:55:16.452842 (+ 287us) client_negotiation.cc:769] Sending connection context
0806 04:55:16.453042 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 04:55:16.453344 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":268,"threads_started":1}
I20260806 04:55:16.455866 30638 heartbeater.cc:347] Connected to a master server at 127.13.148.190:41047
I20260806 04:55:16.456146 30638 heartbeater.cc:464] Registering TS with master...
I20260806 04:55:16.456869 30638 heartbeater.cc:511] Master 127.13.148.190:41047 requested a full tablet report, sending...
I20260806 04:55:16.458288 30318 ts_manager.cc:194] Registered new tserver with Master: e4fda8f8b55d45c9bef0073d6278d7f3 (127.13.148.130:46439)
I20260806 04:55:16.458984 30318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:47031
W20260806 04:55:16.742548 30641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:55:16.742770 30641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:55:16.742849 30641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:55:16.749617 30641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:55:16.749733 30641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:55:16.749898 30641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:55:16.758951 30641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:44299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=44239
--tserver_master_addrs=127.13.148.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:55:16.760057 30641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:55:16.761300 30641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:55:16.773161 30647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:16.773231 30649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:16.773200 30646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:55:16.774188 30641 server_base.cc:1034] running on GCE node
I20260806 04:55:16.774808 30641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:55:16.775674 30641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:55:16.776868 30641 hybrid_clock.cc:648] HybridClock initialized: now 1785992116776828 us; error 45 us; skew 500 ppm
I20260806 04:55:16.777163 30641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:55:16 dist-test-slave-1x4g krb5kdc[29722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992116, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:55:16.779946 30641 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:55:16.781224 30641 webserver.cc:460] Webserver started at http://127.13.148.131:44239/ using document root <none> and password file <none>
I20260806 04:55:16.781809 30641 fs_manager.cc:359] Metadata directory not provided
I20260806 04:55:16.781956 30641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:55:16.787125 30641 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:55:16.789517 30656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:16.790681 30641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:55:16.790835 30641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "4475889dd7e04809864df3a223749914"
format_stamp: "Formatted at 2026-08-06 04:54:54 on dist-test-slave-1x4g"
server_key: "495df165ed8cda7339a0281786981702"
server_key_iv: "e44c63f39f500e85333767016b74d0db"
server_key_version: "encryptionkey@0"
I20260806 04:55:16.791666 30641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:55:16.815161 30641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:55:16.818655 30641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:55:16.818905 30641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:55:16.819694 30641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:55:16.820809 30641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:55:16.820909 30641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:16.821019 30641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:55:16.821102 30641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:16.855552 30641 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:44299
I20260806 04:55:16.855621 30774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:44299 every 8 connection(s)
I20260806 04:55:16.857034 30641 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:55:16.865653 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 30641
Aug 06 04:55:16 dist-test-slave-1x4g krb5kdc[29722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992116, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:55:16.876255 30506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:16.859555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:54109 (local address 127.13.148.190:41047)
0806 04:55:16.859732 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:16.859736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:16.860831 (+ 1095us) server_negotiation.cc:407] Connection header received
0806 04:55:16.862435 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:16.862439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:16.862500 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:16.862580 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:16.863893 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.864614 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.866193 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.866629 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.866784 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:16.870217 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:16.870221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:16.870223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:16.870264 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:16.873069 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:16.873877 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:16.873880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:16.873882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:16.873998 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:16.874501 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:16.874504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:16.874506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:16.874727 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:16.874905 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:16.875634 (+ 729us) server_negotiation.cc:299] Negotiation successful
0806 04:55:16.875888 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 04:55:16.876986 30777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:16.859999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:54109)
0806 04:55:16.860676 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:16.860709 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 04:55:16.862211 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:55:16.862761 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:55:16.862768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:55:16.863263 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:55:16.863759 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:16.863775 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.864746 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.864750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:16.866057 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:16.866068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.867110 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.867115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:16.867316 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:16.868003 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:55:16.868022 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:55:16.870035 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:55:16.873189 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:55:16.873197 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:55:16.873211 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:55:16.873769 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:55:16.874103 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:55:16.874109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:55:16.874113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:55:16.874351 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:55:16.874888 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:55:16.874893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:55:16.875165 (+ 272us) client_negotiation.cc:769] Sending connection context
0806 04:55:16.875451 (+ 286us) client_negotiation.cc:240] Negotiation successful
0806 04:55:16.875855 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":259,"threads_started":1}
I20260806 04:55:16.878350 30775 heartbeater.cc:347] Connected to a master server at 127.13.148.190:41047
I20260806 04:55:16.878641 30775 heartbeater.cc:464] Registering TS with master...
I20260806 04:55:16.879406 30775 heartbeater.cc:511] Master 127.13.148.190:41047 requested a full tablet report, sending...
I20260806 04:55:16.880525 30778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:16.866512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51498 (local address 127.13.148.190:41047)
0806 04:55:16.867034 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:16.867039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:16.867316 (+ 277us) server_negotiation.cc:407] Connection header received
0806 04:55:16.867438 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:16.867442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:16.867516 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:16.867621 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:16.869212 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.869893 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.871194 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.871835 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.871956 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:16.874887 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:16.874893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:16.874897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:16.874930 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:16.877913 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:16.878768 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:16.878776 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:16.878778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:16.878892 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:16.879273 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:16.879275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:16.879277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:16.879505 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:16.879758 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:16.880038 (+ 280us) server_negotiation.cc:299] Negotiation successful
0806 04:55:16.880210 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":255,"threads_started":1}
I20260806 04:55:16.880959 30318 ts_manager.cc:194] Registered new tserver with Master: 4475889dd7e04809864df3a223749914 (127.13.148.131:44299)
I20260806 04:55:16.881845 30318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:54109
I20260806 04:55:16.882893 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:55:16.888762 30778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:16.883741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.13.148.190:41047)
0806 04:55:16.883917 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:16.883925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:16.884357 (+ 432us) server_negotiation.cc:407] Connection header received
0806 04:55:16.884505 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:16.884508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:16.884616 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:16.884724 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:55:16.885167 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.885804 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.887227 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.887649 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.887769 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:16.888169 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:55:16.888330 (+ 161us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 04:55:16.888586 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51506: token expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 04:55:16.888911 30778 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51506: token expired
W20260806 04:55:16.889434 30239 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.13.148.190:41047)
I20260806 04:55:16.900892 30778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:16.889909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.13.148.190:41047)
0806 04:55:16.890046 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:16.890051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:16.890066 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:55:16.890108 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:16.890112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:16.890167 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:16.890250 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:16.891390 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.892015 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.893275 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.893636 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.893767 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:16.896375 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:16.896380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:16.896384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:16.896421 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:16.898642 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:16.899229 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:16.899233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:16.899236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:16.899349 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:16.899671 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:16.899676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:16.899679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:16.899932 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:16.900154 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:16.900408 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 04:55:16.900602 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:55:16.903538 30317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:55:16.910359 30317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:55:16.933766 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:16.921176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36149 (local address 127.0.0.1:50328)
0806 04:55:16.922172 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:16.922204 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 04:55:16.922351 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:55:16.924209 (+ 1858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:55:16.924219 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:55:16.924260 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:55:16.924843 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:16.924859 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.927581 (+ 2722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.927589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:16.930994 (+ 3405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:16.931013 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.932812 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.932820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:16.933029 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:16.933104 (+ 75us) client_negotiation.cc:769] Sending connection context
0806 04:55:16.933232 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 04:55:16.933395 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":2,"thread_start_us":656,"threads_started":1}
I20260806 04:55:16.934201 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:16.921372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50328 (local address 127.13.148.129:36149)
0806 04:55:16.923393 (+ 2021us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:16.923402 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:16.923447 (+ 45us) server_negotiation.cc:407] Connection header received
0806 04:55:16.923559 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:16.923568 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:16.923943 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:16.924189 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:55:16.925113 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.927385 (+ 2272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.931182 (+ 3797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.932556 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.932713 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:16.932911 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:16.933269 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 04:55:16.933409 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1865,"thread_start_us":1773,"threads_started":1}
I20260806 04:55:16.937996 30438 tablet_service.cc:1467] Processing CreateTablet for tablet 526ea6ad64024c09b5a4d6513404fc02 (DEFAULT_TABLE table=test-table [id=7b4b3d25689140d4844733dc36a34a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:55:16.939348 30791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:16.923740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.13.148.131:44299)
0806 04:55:16.925251 (+ 1511us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:16.925263 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:16.931715 (+ 6452us) server_negotiation.cc:407] Connection header received
0806 04:55:16.931959 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:16.931964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:16.932121 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:16.932301 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:55:16.932903 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.934657 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.936566 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.937699 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.937885 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:16.938052 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:16.938838 (+ 786us) server_negotiation.cc:299] Negotiation successful
0806 04:55:16.939007 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1313,"thread_start_us":1073,"threads_started":1}
I20260806 04:55:16.939405 30789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:16.923204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:44299 (local address 127.0.0.1:40046)
0806 04:55:16.931619 (+ 8415us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:16.931634 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:55:16.931814 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:55:16.932290 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:55:16.932295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:55:16.932318 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:55:16.932664 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:16.932676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.934794 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.934799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:16.936409 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:16.936420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.938206 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.938212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:16.938379 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:16.938413 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 04:55:16.938501 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 04:55:16.938627 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8275,"mutex_wait_us":62,"thread_start_us":1254,"threads_started":1}
I20260806 04:55:16.940174 30790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:16.921502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:46439 (local address 127.0.0.1:52876)
0806 04:55:16.923804 (+ 2302us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:16.923824 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:55:16.923977 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:55:16.932596 (+ 8619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:55:16.932602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:55:16.932629 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:55:16.933146 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:16.933159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.935134 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.935139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:16.937200 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:16.937215 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.939604 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.939612 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:16.939755 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:16.939777 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:55:16.939860 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 04:55:16.939937 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2136,"mutex_wait_us":26,"thread_start_us":9811,"threads_started":1}
I20260806 04:55:16.940424 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:16.921557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.13.148.130:46439)
0806 04:55:16.931843 (+ 10286us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:16.931850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:16.931883 (+ 33us) server_negotiation.cc:407] Connection header received
0806 04:55:16.931967 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:16.931974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:16.932184 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:16.932375 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:55:16.933351 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.934998 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.937370 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:16.939066 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:16.939257 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:16.939426 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:16.939932 (+ 506us) server_negotiation.cc:299] Negotiation successful
0806 04:55:16.940088 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10139,"thread_start_us":10034,"threads_started":1}
I20260806 04:55:16.941020 30438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526ea6ad64024c09b5a4d6513404fc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:55:16.942377 30709 tablet_service.cc:1467] Processing CreateTablet for tablet 526ea6ad64024c09b5a4d6513404fc02 (DEFAULT_TABLE table=test-table [id=7b4b3d25689140d4844733dc36a34a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:55:16.943688 30709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526ea6ad64024c09b5a4d6513404fc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:55:16.946441 30571 tablet_service.cc:1467] Processing CreateTablet for tablet 526ea6ad64024c09b5a4d6513404fc02 (DEFAULT_TABLE table=test-table [id=7b4b3d25689140d4844733dc36a34a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:55:16.947774 30571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526ea6ad64024c09b5a4d6513404fc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:55:16.954532 30792 tablet_bootstrap.cc:492] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914: Bootstrap starting.
I20260806 04:55:16.956842 30793 tablet_bootstrap.cc:492] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063: Bootstrap starting.
I20260806 04:55:16.959136 30792 tablet_bootstrap.cc:654] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914: Neither blocks nor log segments found. Creating new log.
I20260806 04:55:16.960322 30793 tablet_bootstrap.cc:654] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063: Neither blocks nor log segments found. Creating new log.
I20260806 04:55:16.960391 30792 log.cc:824] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914: Log is configured to *not* fsync() on all Append() calls
I20260806 04:55:16.961562 30793 log.cc:824] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063: Log is configured to *not* fsync() on all Append() calls
I20260806 04:55:16.962670 30795 tablet_bootstrap.cc:492] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3: Bootstrap starting.
I20260806 04:55:16.963321 30792 tablet_bootstrap.cc:492] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914: No bootstrap required, opened a new log
I20260806 04:55:16.963575 30792 ts_tablet_manager.cc:1397] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 04:55:16.967039 30795 tablet_bootstrap.cc:654] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3: Neither blocks nor log segments found. Creating new log.
I20260806 04:55:16.968323 30795 log.cc:824] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3: Log is configured to *not* fsync() on all Append() calls
I20260806 04:55:16.969800 30793 tablet_bootstrap.cc:492] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063: No bootstrap required, opened a new log
I20260806 04:55:16.970064 30793 ts_tablet_manager.cc:1397] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260806 04:55:16.970741 30795 tablet_bootstrap.cc:492] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3: No bootstrap required, opened a new log
I20260806 04:55:16.970969 30795 ts_tablet_manager.cc:1397] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260806 04:55:16.972607 30792 raft_consensus.cc:348] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } }
I20260806 04:55:16.973034 30792 raft_consensus.cc:374] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:55:16.973136 30792 raft_consensus.cc:729] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4475889dd7e04809864df3a223749914, State: Initialized, Role: FOLLOWER
I20260806 04:55:16.973755 30792 consensus_queue.cc:260] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } }
I20260806 04:55:16.974896 30775 heartbeater.cc:503] Master 127.13.148.190:41047 was elected leader, sending a full tablet report...
I20260806 04:55:16.975574 30792 ts_tablet_manager.cc:1428] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260806 04:55:16.979143 30795 raft_consensus.cc:348] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } }
I20260806 04:55:16.979568 30795 raft_consensus.cc:374] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:55:16.979672 30795 raft_consensus.cc:729] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4fda8f8b55d45c9bef0073d6278d7f3, State: Initialized, Role: FOLLOWER
I20260806 04:55:16.980152 30793 raft_consensus.cc:348] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } }
I20260806 04:55:16.980250 30795 consensus_queue.cc:260] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } }
I20260806 04:55:16.980480 30793 raft_consensus.cc:374] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:55:16.980574 30793 raft_consensus.cc:729] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d1b567085d45a080c2dda2c9bdb063, State: Initialized, Role: FOLLOWER
I20260806 04:55:16.981180 30793 consensus_queue.cc:260] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } }
I20260806 04:55:16.981783 30638 heartbeater.cc:503] Master 127.13.148.190:41047 was elected leader, sending a full tablet report...
I20260806 04:55:16.982146 30504 heartbeater.cc:503] Master 127.13.148.190:41047 was elected leader, sending a full tablet report...
I20260806 04:55:16.982765 30793 ts_tablet_manager.cc:1428] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260806 04:55:16.983385 30795 ts_tablet_manager.cc:1428] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
W20260806 04:55:17.028087 30505 tablet.cc:2369] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:55:17.108645 30776 tablet.cc:2369] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:55:17.158192 30799 raft_consensus.cc:482] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:55:17.158509 30799 raft_consensus.cc:504] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } }
I20260806 04:55:17.160096 30799 leader_election.cc:288] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36d1b567085d45a080c2dda2c9bdb063 (127.13.148.129:36149), 4475889dd7e04809864df3a223749914 (127.13.148.131:44299)
I20260806 04:55:17.166769 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:17.160366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:37517 (local address 127.13.148.129:36149)
0806 04:55:17.160582 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:17.160587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:17.161214 (+ 627us) server_negotiation.cc:407] Connection header received
0806 04:55:17.161400 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:17.161406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:17.161491 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:17.161587 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:55:17.162253 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:17.163152 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:17.164921 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:17.165837 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:17.165942 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:17.165988 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:17.166399 (+ 411us) server_negotiation.cc:299] Negotiation successful
0806 04:55:17.166527 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 04:55:17.166797 30801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:17.160438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36149 (local address 127.13.148.130:37517)
0806 04:55:17.161106 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:17.161120 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:55:17.161268 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:55:17.161683 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:55:17.161688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:55:17.161726 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:55:17.162132 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:17.162144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:17.163285 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:17.163290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:17.164729 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:17.164741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:17.166064 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:17.166071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:17.166245 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:17.166284 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 04:55:17.166349 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 04:55:17.166444 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":394,"threads_started":1}
I20260806 04:55:17.166821 30791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:17.161433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:44109 (local address 127.13.148.131:44299)
0806 04:55:17.161653 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:17.161658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:17.161882 (+ 224us) server_negotiation.cc:407] Connection header received
0806 04:55:17.162048 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:17.162055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:17.162131 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:17.162219 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:55:17.162758 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:17.163517 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:17.165216 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:17.166025 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:17.166166 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:17.166230 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:17.166494 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 04:55:17.166617 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 04:55:17.166992 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:17.161187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:44299 (local address 127.13.148.130:44109)
0806 04:55:17.161784 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:17.161800 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:55:17.161918 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:55:17.162246 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:55:17.162250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:55:17.162271 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:55:17.162618 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:17.162630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:17.163637 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:17.163640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:17.165071 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:17.165085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:17.166205 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:17.166209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:17.166319 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:17.166365 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 04:55:17.166445 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 04:55:17.166540 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":297,"threads_started":1}
I20260806 04:55:17.167659 30459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526ea6ad64024c09b5a4d6513404fc02" candidate_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d1b567085d45a080c2dda2c9bdb063" is_pre_election: true
I20260806 04:55:17.167846 30729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526ea6ad64024c09b5a4d6513404fc02" candidate_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4475889dd7e04809864df3a223749914" is_pre_election: true
I20260806 04:55:17.168149 30459 raft_consensus.cc:2454] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4fda8f8b55d45c9bef0073d6278d7f3 in term 0.
I20260806 04:55:17.168318 30729 raft_consensus.cc:2454] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4fda8f8b55d45c9bef0073d6278d7f3 in term 0.
I20260806 04:55:17.168927 30527 leader_election.cc:302] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d1b567085d45a080c2dda2c9bdb063, e4fda8f8b55d45c9bef0073d6278d7f3; no voters:
I20260806 04:55:17.169256 30799 raft_consensus.cc:2781] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:55:17.169405 30799 raft_consensus.cc:482] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:55:17.169509 30799 raft_consensus.cc:3037] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:55:17.171609 30799 raft_consensus.cc:504] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } }
I20260806 04:55:17.172137 30799 leader_election.cc:288] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [CANDIDATE]: Term 1 election: Requested vote from peers 36d1b567085d45a080c2dda2c9bdb063 (127.13.148.129:36149), 4475889dd7e04809864df3a223749914 (127.13.148.131:44299)
I20260806 04:55:17.172519 30459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526ea6ad64024c09b5a4d6513404fc02" candidate_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d1b567085d45a080c2dda2c9bdb063"
I20260806 04:55:17.172608 30729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526ea6ad64024c09b5a4d6513404fc02" candidate_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4475889dd7e04809864df3a223749914"
I20260806 04:55:17.172695 30459 raft_consensus.cc:3037] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:55:17.172812 30729 raft_consensus.cc:3037] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:55:17.174930 30459 raft_consensus.cc:2454] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4fda8f8b55d45c9bef0073d6278d7f3 in term 1.
I20260806 04:55:17.175047 30729 raft_consensus.cc:2454] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4fda8f8b55d45c9bef0073d6278d7f3 in term 1.
I20260806 04:55:17.175312 30527 leader_election.cc:302] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d1b567085d45a080c2dda2c9bdb063, e4fda8f8b55d45c9bef0073d6278d7f3; no voters:
I20260806 04:55:17.175592 30799 raft_consensus.cc:2781] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:55:17.176061 30799 raft_consensus.cc:686] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 1 LEADER]: Becoming Leader. State: Replica: e4fda8f8b55d45c9bef0073d6278d7f3, State: Running, Role: LEADER
I20260806 04:55:17.176519 30799 consensus_queue.cc:237] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } } peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } }
I20260806 04:55:17.181164 30317 catalog_manager.cc:5495] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4fda8f8b55d45c9bef0073d6278d7f3 (127.13.148.130). New cstate: current_term: 1 leader_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4fda8f8b55d45c9bef0073d6278d7f3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 } health_report { overall_health: UNKNOWN } } }
W20260806 04:55:17.186215 30639 tablet.cc:2369] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:55:17.228744 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:17.223591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.13.148.130:46439)
0806 04:55:17.223774 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:17.223781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:17.223857 (+ 76us) server_negotiation.cc:407] Connection header received
0806 04:55:17.224084 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:17.224088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:17.224158 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:17.224300 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:55:17.224751 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:17.225497 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:17.226800 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:17.227220 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:17.227339 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:17.227798 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:55:17.228226 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:55:17.228367 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:17.228417 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 04:55:17.228486 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:55:17.236506 30729 raft_consensus.cc:1261] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914 [term 1 FOLLOWER]: Refusing update from remote peer e4fda8f8b55d45c9bef0073d6278d7f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:55:17.236518 30459 raft_consensus.cc:1261] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063 [term 1 FOLLOWER]: Refusing update from remote peer e4fda8f8b55d45c9bef0073d6278d7f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:55:17.237519 30803 consensus_queue.cc:1035] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d1b567085d45a080c2dda2c9bdb063" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:55:17.237913 30799 consensus_queue.cc:1035] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4475889dd7e04809864df3a223749914" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 44299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:55:17.262980 30317 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:55:17.263202 30317 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:55:17.266474 30317 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Sending DeleteTablet for 3 replicas of tablet 526ea6ad64024c09b5a4d6513404fc02
I20260806 04:55:17.267532 30571 tablet_service.cc:1514] Processing DeleteTablet for tablet 526ea6ad64024c09b5a4d6513404fc02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:55:17 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:52876
I20260806 04:55:17.267573 30709 tablet_service.cc:1514] Processing DeleteTablet for tablet 526ea6ad64024c09b5a4d6513404fc02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:55:17 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:40046
I20260806 04:55:17.268348 30816 tablet_replica.cc:330] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3: stopping tablet replica
I20260806 04:55:17.268589 30438 tablet_service.cc:1514] Processing DeleteTablet for tablet 526ea6ad64024c09b5a4d6513404fc02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:55:17 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:50328
I20260806 04:55:17.268842 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 30355
I20260806 04:55:17.268865 30816 raft_consensus.cc:2229] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:55:17.281219 30816 raft_consensus.cc:2258] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:55:17.281311 30818 tablet_replica.cc:330] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063: stopping tablet replica
I20260806 04:55:17.281800 30818 raft_consensus.cc:2229] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:55:17.282371 30818 raft_consensus.cc:2258] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:55:17.283361 30816 ts_tablet_manager.cc:1905] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:55:17.288225 30818 ts_tablet_manager.cc:1905] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:55:17.289353 30817 tablet_replica.cc:330] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914: stopping tablet replica
I20260806 04:55:17.289709 30817 raft_consensus.cc:2229] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:55:17.290145 30817 raft_consensus.cc:2258] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:55:17.291766 30816 ts_tablet_manager.cc:1918] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:55:17.291927 30816 log.cc:1196] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/526ea6ad64024c09b5a4d6513404fc02
I20260806 04:55:17.292448 30816 ts_tablet_manager.cc:1939] T 526ea6ad64024c09b5a4d6513404fc02 P e4fda8f8b55d45c9bef0073d6278d7f3: Deleting consensus metadata
I20260806 04:55:17.294109 30303 catalog_manager.cc:4841] TS e4fda8f8b55d45c9bef0073d6278d7f3 (127.13.148.130:46439): tablet 526ea6ad64024c09b5a4d6513404fc02 (table test-table [id=7b4b3d25689140d4844733dc36a34a67]) successfully deleted
I20260806 04:55:17.294376 30818 ts_tablet_manager.cc:1918] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:55:17.294633 30818 log.cc:1196] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/526ea6ad64024c09b5a4d6513404fc02
I20260806 04:55:17.295650 30818 ts_tablet_manager.cc:1939] T 526ea6ad64024c09b5a4d6513404fc02 P 36d1b567085d45a080c2dda2c9bdb063: Deleting consensus metadata
I20260806 04:55:17.296087 30817 ts_tablet_manager.cc:1905] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:55:17.297726 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:17.269816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50330 (local address 127.13.148.129:36149)
0806 04:55:17.270518 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:17.270526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:17.270546 (+ 20us) server_negotiation.cc:407] Connection header received
0806 04:55:17.270607 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:17.270614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:17.270711 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:17.270809 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:17.273249 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:17.274723 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:17.277329 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:17.278190 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:17.278327 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:17.282312 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:17.282327 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:17.282344 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:17.282384 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:17.287645 (+ 5261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:17.294807 (+ 7162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:17.294817 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:17.294820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:17.294974 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:17.295479 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:17.295485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:17.295489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:17.296206 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:17.296508 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:17.297135 (+ 627us) server_negotiation.cc:299] Negotiation successful
0806 04:55:17.297377 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545}
I20260806 04:55:17.297713 30305 catalog_manager.cc:4841] TS 36d1b567085d45a080c2dda2c9bdb063 (127.13.148.129:36149): tablet 526ea6ad64024c09b5a4d6513404fc02 (table test-table [id=7b4b3d25689140d4844733dc36a34a67]) successfully deleted
I20260806 04:55:17.299336 30499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:55:17.304078 30817 ts_tablet_manager.cc:1918] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:55:17.304245 30817 log.cc:1196] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/526ea6ad64024c09b5a4d6513404fc02
I20260806 04:55:17.304996 30817 ts_tablet_manager.cc:1939] T 526ea6ad64024c09b5a4d6513404fc02 P 4475889dd7e04809864df3a223749914: Deleting consensus metadata
I20260806 04:55:17.306797 30305 catalog_manager.cc:4841] TS 4475889dd7e04809864df3a223749914 (127.13.148.131:44299): tablet 526ea6ad64024c09b5a4d6513404fc02 (table test-table [id=7b4b3d25689140d4844733dc36a34a67]) successfully deleted
I20260806 04:55:17.387888 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 30355
I20260806 04:55:17.412719 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 30508
I20260806 04:55:17.427915 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:17.413559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52902 (local address 127.13.148.130:46439)
0806 04:55:17.413735 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:17.413741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:17.413764 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:55:17.413872 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:17.413878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:17.413955 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:17.414073 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:17.415375 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:17.416533 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:17.417949 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:17.418600 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:17.418726 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:17.421090 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:17.421104 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:17.421142 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:17.421191 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:17.425241 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:17.425908 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:17.425917 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:17.425921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:17.426108 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:17.426420 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:17.426425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:17.426428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:17.426848 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:17.427151 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:17.427299 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 04:55:17.427513 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 04:55:17.428628 30632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:55:17.531545 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 30508
I20260806 04:55:17.551481 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 30641
I20260806 04:55:17.565862 30791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:17.552181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40062 (local address 127.13.148.131:44299)
0806 04:55:17.552369 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:17.552375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:17.552393 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:55:17.552549 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:17.552555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:17.552625 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:17.552730 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:17.553999 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:17.555164 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:17.556508 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:17.557213 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:17.557332 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:17.559875 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:17.559891 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:17.559910 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:17.559947 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:17.563217 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:17.563829 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:17.563839 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:17.563842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:17.564005 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:17.564306 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:17.564310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:17.564314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:17.564743 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:17.565064 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:17.565297 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 04:55:17.565506 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 04:55:17.566483 30770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:55:17.647053 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 30641
I20260806 04:55:17.664233 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 30284
I20260806 04:55:17.665423 30348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:55:17.770583 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 30284
I20260806 04:55:37.792891 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41047
--webserver_interface=127.13.148.190
--webserver_port=40965
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:41047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:55:38.095717 30823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:55:38.095991 30823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:55:38.096143 30823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:55:38.102604 30823 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 04:55:38.102679 30823 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 04:55:38.102712 30823 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 04:55:38.102749 30823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:55:38.102823 30823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:55:38.102913 30823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:55:38.102969 30823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:55:38.103056 30823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:55:38.111964 30823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:41047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=40965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:55:38.113109 30823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:55:38.114301 30823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:55:38.125001 30829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:38.125236 30831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:38.124984 30828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:55:38.126168 30823 server_base.cc:1034] running on GCE node
I20260806 04:55:38.126902 30823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:55:38.128091 30823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:55:38.129313 30823 hybrid_clock.cc:648] HybridClock initialized: now 1785992138129247 us; error 41 us; skew 500 ppm
I20260806 04:55:38.129653 30823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:55:38 dist-test-slave-1x4g krb5kdc[29722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992138, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:55:38.132555 30823 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:55:38.133946 30823 webserver.cc:460] Webserver started at http://127.13.148.190:40965/ using document root <none> and password file <none>
I20260806 04:55:38.134635 30823 fs_manager.cc:359] Metadata directory not provided
I20260806 04:55:38.134769 30823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:55:38.139735 30823 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 04:55:38.142347 30838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:38.143576 30823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:55:38.143731 30823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "a79f52b5e4274220b876f5422483397c"
format_stamp: "Formatted at 2026-08-06 04:54:52 on dist-test-slave-1x4g"
server_key: "6e2608cab18140526aa8ee3c77fdd452"
server_key_iv: "26555c209b659eeaacfced78e4055c91"
server_key_version: "encryptionkey@0"
I20260806 04:55:38.144649 30823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:55:38.174790 30823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:55:38.178408 30823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:55:38.178690 30823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:55:38.195621 30823 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:41047
I20260806 04:55:38.195652 30889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:41047 every 8 connection(s)
I20260806 04:55:38.197119 30823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:55:38.198411 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 30823
I20260806 04:55:38.199482 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:36149
--local_ip_for_outbound_sockets=127.13.148.129
--tserver_master_addrs=127.13.148.190:41047
--webserver_port=40687
--webserver_interface=127.13.148.129
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:55:38.206640 30890 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:55:38.211105 30890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Bootstrap starting.
I20260806 04:55:38.229816 30890 log.cc:824] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:55:38.244396 30890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:55:38.244925 30890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Bootstrap complete.
I20260806 04:55:38.253566 30890 raft_consensus.cc:348] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } }
I20260806 04:55:38.254352 30890 raft_consensus.cc:729] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a79f52b5e4274220b876f5422483397c, State: Initialized, Role: FOLLOWER
I20260806 04:55:38.254917 30890 consensus_queue.cc:260] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } }
I20260806 04:55:38.255174 30890 raft_consensus.cc:388] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:55:38.255285 30890 raft_consensus.cc:482] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:55:38.255447 30890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:55:38.258934 30890 raft_consensus.cc:504] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } }
I20260806 04:55:38.259357 30890 leader_election.cc:302] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a79f52b5e4274220b876f5422483397c; no voters:
I20260806 04:55:38.259974 30890 leader_election.cc:288] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 04:55:38.260092 30894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:55:38.261247 30894 raft_consensus.cc:686] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [term 3 LEADER]: Becoming Leader. State: Replica: a79f52b5e4274220b876f5422483397c, State: Running, Role: LEADER
I20260806 04:55:38.261832 30894 consensus_queue.cc:237] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } }
I20260806 04:55:38.262030 30890 sys_catalog.cc:564] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:55:38.263952 30896 sys_catalog.cc:455] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a79f52b5e4274220b876f5422483397c. Latest consensus state: current_term: 3 leader_uuid: "a79f52b5e4274220b876f5422483397c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } } }
I20260806 04:55:38.264328 30896 sys_catalog.cc:458] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [sys.catalog]: This master's current role is: LEADER
I20260806 04:55:38.264626 30895 sys_catalog.cc:455] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a79f52b5e4274220b876f5422483397c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79f52b5e4274220b876f5422483397c" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41047 } } }
I20260806 04:55:38.264984 30895 sys_catalog.cc:458] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c [sys.catalog]: This master's current role is: LEADER
I20260806 04:55:38.276151 30903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:55:38.284386 30903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:55:38.285754 30903 catalog_manager.cc:1259] Loaded cluster ID: ce1261c06295411fbb437b0bce6b5f4d
I20260806 04:55:38.285885 30903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:55:38.289839 30903 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:55:38.291391 30903 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Loaded TSK: 0
I20260806 04:55:38.291581 30903 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a79f52b5e4274220b876f5422483397c: Loaded TSK: 1
F20260806 04:55:38.292559 30903 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f22484779ad google::LogMessage::Fail() at ??:0
@ 0x7f224847a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f22484774de google::LogMessage::Flush() at ??:0
@ 0x7f2248478a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f224c1db7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f224c1ca73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f225dea61c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f225dea5783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f225df312a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f225df310c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f225df7b5e9 std::function<>::operator()() at ??:0
@ 0x7f225dea9cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f225dea7c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f225df2f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f225df2f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f224b0c3345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f224b0c691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f224b0c6799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f224b0a1eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f224e37b6db start_thread at ??:0
@ 0x7f224731161f clone at ??:0
W20260806 04:55:38.549126 30892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:55:38.549404 30892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:55:38.549554 30892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:55:38.556521 30892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:55:38.556671 30892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:55:38.556903 30892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:55:38.565106 30892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:36149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=40687
--tserver_master_addrs=127.13.148.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:55:38.566280 30892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:55:38.567554 30892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:55:38.580130 30939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:38.580160 30942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:38.580191 30940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:55:38.580924 30892 server_base.cc:1034] running on GCE node
I20260806 04:55:38.581465 30892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:55:38.582286 30892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:55:38.583494 30892 hybrid_clock.cc:648] HybridClock initialized: now 1785992138583457 us; error 48 us; skew 500 ppm
I20260806 04:55:38.583770 30892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:55:38 dist-test-slave-1x4g krb5kdc[29722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992138, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:55:38.586668 30892 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:55:38.588033 30892 webserver.cc:460] Webserver started at http://127.13.148.129:40687/ using document root <none> and password file <none>
I20260806 04:55:38.588620 30892 fs_manager.cc:359] Metadata directory not provided
I20260806 04:55:38.588752 30892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:55:38.593667 30892 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260806 04:55:38.596194 30949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:38.597520 30892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:55:38.597699 30892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "36d1b567085d45a080c2dda2c9bdb063"
format_stamp: "Formatted at 2026-08-06 04:54:53 on dist-test-slave-1x4g"
server_key: "8f32e550d2cbbd33a9648bfcb0825a8a"
server_key_iv: "0f95f0c46d3b4b86102bc3c2b2d216ff"
server_key_version: "encryptionkey@0"
I20260806 04:55:38.598634 30892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:55:38.615196 30892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:55:38.618834 30892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:55:38.619096 30892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:55:38.620074 30892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:55:38.621272 30892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:55:38.621394 30892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:38.621520 30892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:55:38.621614 30892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:38.653489 30892 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:36149
I20260806 04:55:38.653577 31063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:36149 every 8 connection(s)
I20260806 04:55:38.654873 30892 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:55:38.656661 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 30892
I20260806 04:55:38.657445 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:46439
--local_ip_for_outbound_sockets=127.13.148.130
--tserver_master_addrs=127.13.148.190:41047
--webserver_port=38069
--webserver_interface=127.13.148.130
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:55:38.659968 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:38.658096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:48815)
0806 04:55:38.658886 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:38.659156 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":341,"threads_started":1}
W20260806 04:55:38.660665 31065 heartbeater.cc:644] Failed to heartbeat to 127.13.148.190:41047 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.190:41047: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
I20260806 04:55:38.661479 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:38.661174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:36743)
0806 04:55:38.661311 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:38.661383 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 04:55:38.662367 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:38.662045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:38877)
0806 04:55:38.662170 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:38.662252 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 04:55:38.662652 31065 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 04:55:38.966888 31069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:55:38.967155 31069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:55:38.967252 31069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:55:38.973166 31069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:55:38.973249 31069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:55:38.973340 31069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:55:38.981391 31069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:46439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=38069
--tserver_master_addrs=127.13.148.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:55:38.982375 31069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:55:38.983443 31069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:55:38.995002 31075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:38.995071 31076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:38.995458 31078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:55:38.996034 31069 server_base.cc:1034] running on GCE node
I20260806 04:55:38.996608 31069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:55:38.997908 31069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:55:38.999122 31069 hybrid_clock.cc:648] HybridClock initialized: now 1785992138999081 us; error 53 us; skew 500 ppm
I20260806 04:55:38.999403 31069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:55:39 dist-test-slave-1x4g krb5kdc[29722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992139, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:55:39.006537 31069 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:55:39.007916 31069 webserver.cc:460] Webserver started at http://127.13.148.130:38069/ using document root <none> and password file <none>
I20260806 04:55:39.008544 31069 fs_manager.cc:359] Metadata directory not provided
I20260806 04:55:39.008682 31069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:55:39.013584 31069 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:55:39.018138 31086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:39.019331 31069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:55:39.019500 31069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "e4fda8f8b55d45c9bef0073d6278d7f3"
format_stamp: "Formatted at 2026-08-06 04:54:53 on dist-test-slave-1x4g"
server_key: "1c47c0ebfec6bde4c167d911a546e37c"
server_key_iv: "8c571feac16eda73ab981e7544f9fc77"
server_key_version: "encryptionkey@0"
I20260806 04:55:39.020481 31069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:55:39.038856 31069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:55:39.042423 31069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:55:39.042672 31069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:55:39.043546 31069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:55:39.044741 31069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:55:39.044888 31069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:39.045019 31069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:55:39.045100 31069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:39.076344 31069 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:46439
I20260806 04:55:39.076427 31200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:46439 every 8 connection(s)
I20260806 04:55:39.077840 31069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:55:39.081734 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 31069
I20260806 04:55:39.082268 31203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:39.080537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:40661)
0806 04:55:39.081261 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:39.081464 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":282,"threads_started":1}
I20260806 04:55:39.082293 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:44299
--local_ip_for_outbound_sockets=127.13.148.131
--tserver_master_addrs=127.13.148.190:41047
--webserver_port=44239
--webserver_interface=127.13.148.131
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:55:39.082973 31201 heartbeater.cc:644] Failed to heartbeat to 127.13.148.190:41047 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.190:41047: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
I20260806 04:55:39.083745 31203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:39.083435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:56067)
0806 04:55:39.083576 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:39.083650 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 04:55:39.084537 31203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:39.084234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:44815)
0806 04:55:39.084363 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:39.084440 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 04:55:39.084771 31201 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 04:55:39.388020 31204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:55:39.388291 31204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:55:39.388441 31204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:55:39.394832 31204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:55:39.394958 31204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:55:39.395133 31204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:55:39.403743 31204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:44299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=44239
--tserver_master_addrs=127.13.148.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:55:39.404960 31204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:55:39.406265 31204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:55:39.418637 31210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:39.418561 31213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:39.418612 31211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:55:39.419669 31204 server_base.cc:1034] running on GCE node
I20260806 04:55:39.420305 31204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:55:39.421317 31204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:55:39.422497 31204 hybrid_clock.cc:648] HybridClock initialized: now 1785992139422465 us; error 30 us; skew 500 ppm
I20260806 04:55:39.422770 31204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:55:39 dist-test-slave-1x4g krb5kdc[29722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992139, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:55:39.425698 31204 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:55:39.427121 31204 webserver.cc:460] Webserver started at http://127.13.148.131:44239/ using document root <none> and password file <none>
I20260806 04:55:39.427757 31204 fs_manager.cc:359] Metadata directory not provided
I20260806 04:55:39.427892 31204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:55:39.432646 31204 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:55:39.435073 31220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:39.436367 31204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:55:39.436568 31204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "4475889dd7e04809864df3a223749914"
format_stamp: "Formatted at 2026-08-06 04:54:54 on dist-test-slave-1x4g"
server_key: "495df165ed8cda7339a0281786981702"
server_key_iv: "e44c63f39f500e85333767016b74d0db"
server_key_version: "encryptionkey@0"
I20260806 04:55:39.437592 31204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:55:39.462397 31204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:55:39.466759 31204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:55:39.467043 31204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:55:39.467964 31204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:55:39.469246 31204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:55:39.469348 31204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:39.469468 31204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:55:39.469564 31204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:39.498889 31204 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:44299
I20260806 04:55:39.498975 31335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:44299 every 8 connection(s)
I20260806 04:55:39.500209 31204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:55:39.504528 31339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:39.502922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:41719)
0806 04:55:39.503556 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:39.503781 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":236,"threads_started":1}
W20260806 04:55:39.505152 31336 heartbeater.cc:644] Failed to heartbeat to 127.13.148.190:41047 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.190:41047: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
I20260806 04:55:39.505780 31339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:39.505529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:37055)
0806 04:55:39.505644 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:39.505698 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 04:55:39.506526 31339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:39.506252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:41715)
0806 04:55:39.506362 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:39.506429 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 04:55:39.506754 31336 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 04:55:39.507921 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 31204
I20260806 04:55:39.663885 31341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:39.663101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:55285)
0806 04:55:39.663539 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:39.663641 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":227,"threads_started":1}
I20260806 04:55:40.086082 31342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:40.085292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:60849)
0806 04:55:40.085726 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:40.085828 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":215,"threads_started":1}
I20260806 04:55:40.507972 31343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:40.507218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:48225)
0806 04:55:40.507667 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:40.507750 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":217,"threads_started":1}
I20260806 04:55:40.665462 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:40.664774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:33951)
0806 04:55:40.665245 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:40.665318 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":230,"threads_started":1}
I20260806 04:55:41.087606 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:41.086924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:38445)
0806 04:55:41.087388 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:41.087460 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":224,"threads_started":1}
I20260806 04:55:41.509745 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:41.509089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:59585)
0806 04:55:41.509525 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:41.509598 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":217,"threads_started":1}
I20260806 04:55:41.666824 31347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:41.666204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:37459)
0806 04:55:41.666597 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:41.666671 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":210,"threads_started":1}
I20260806 04:55:42.089193 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:42.088451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:53719)
0806 04:55:42.088981 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:42.089054 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":262,"threads_started":1}
I20260806 04:55:42.511209 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:42.510482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:36101)
0806 04:55:42.510981 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:42.511062 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":305,"threads_started":1}
I20260806 04:55:42.668257 31350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:42.667651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:44415)
0806 04:55:42.668021 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:42.668090 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":180,"threads_started":1}
I20260806 04:55:43.090613 31351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:43.089978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:36087)
0806 04:55:43.090366 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:43.090438 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":185,"threads_started":1}
I20260806 04:55:43.512750 31352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:43.512020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:51429)
0806 04:55:43.512477 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:43.512588 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":259,"threads_started":1}
I20260806 04:55:43.669786 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:43.669150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:48387)
0806 04:55:43.669569 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:43.669641 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":213,"threads_started":1}
I20260806 04:55:44.092170 31354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:44.091557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:41421)
0806 04:55:44.091945 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:44.092023 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":205,"threads_started":1}
I20260806 04:55:44.514348 31355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:44.513585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:36693)
0806 04:55:44.514118 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:44.514193 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":328,"threads_started":1}
I20260806 04:55:44.671183 31356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:44.670581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:46833)
0806 04:55:44.670969 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:44.671046 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":209,"threads_started":1}
I20260806 04:55:45.093865 31357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:45.093252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:60873)
0806 04:55:45.093650 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:45.093729 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":213,"threads_started":1}
I20260806 04:55:45.516239 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:45.515562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:49415)
0806 04:55:45.516013 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:45.516094 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":247,"threads_started":1}
I20260806 04:55:45.672942 31359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:45.672014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:56533)
0806 04:55:45.672586 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:45.672667 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":283,"threads_started":1}
I20260806 04:55:46.095355 31360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:46.094669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:41073)
0806 04:55:46.095097 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:46.095167 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":198,"threads_started":1}
I20260806 04:55:46.517957 31361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:46.517176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:54489)
0806 04:55:46.517660 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:46.517741 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":216,"threads_started":1}
I20260806 04:55:46.674378 31362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:46.673761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:58313)
0806 04:55:46.674159 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:46.674231 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":205,"threads_started":1}
I20260806 04:55:47.097090 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:47.096207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:44781)
0806 04:55:47.096847 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:47.096931 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":348,"threads_started":1}
I20260806 04:55:47.519405 31364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:47.518809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:40461)
0806 04:55:47.519197 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:47.519267 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":193,"threads_started":1}
I20260806 04:55:47.675930 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:47.675296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:45179)
0806 04:55:47.675705 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:47.675778 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":207,"threads_started":1}
I20260806 04:55:48.098623 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:48.097963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:52965)
0806 04:55:48.098395 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:48.098467 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":226,"threads_started":1}
I20260806 04:55:48.520989 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:48.520312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:43395)
0806 04:55:48.520711 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:48.520833 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":210,"threads_started":1}
I20260806 04:55:48.677490 31368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:48.676801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:43501)
0806 04:55:48.677266 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:48.677342 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":229,"threads_started":1}
I20260806 04:55:49.100343 31369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:49.099589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:40613)
0806 04:55:49.100123 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:49.100195 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":342,"threads_started":1}
I20260806 04:55:49.522406 31371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:49.521773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:38331)
0806 04:55:49.522192 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:49.522267 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":233,"threads_started":1}
I20260806 04:55:49.679059 31372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:49.678332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:59877)
0806 04:55:49.678796 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:49.678880 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":266,"threads_started":1}
I20260806 04:55:50.101930 31373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:50.101203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:38095)
0806 04:55:50.101618 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:50.101702 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":207,"threads_started":1}
I20260806 04:55:50.523880 31374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:50.523219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:41051)
0806 04:55:50.523619 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:50.523702 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":189,"threads_started":1}
I20260806 04:55:50.680624 31375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:50.679897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:55247)
0806 04:55:50.680403 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:50.680480 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":308,"threads_started":1}
I20260806 04:55:51.103546 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:51.102922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:39617)
0806 04:55:51.103315 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:51.103392 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":195,"threads_started":1}
I20260806 04:55:51.525447 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:51.524762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:53517)
0806 04:55:51.525223 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:51.525305 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":259,"threads_started":1}
I20260806 04:55:51.682135 31379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:51.681426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:53099)
0806 04:55:51.681864 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:51.681967 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":241,"threads_started":1}
I20260806 04:55:52.105118 31380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:52.104383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:51855)
0806 04:55:52.104859 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:52.104955 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":202,"threads_started":1}
I20260806 04:55:52.527100 31381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:52.526281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:39863)
0806 04:55:52.526789 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:52.526886 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":301,"threads_started":1}
I20260806 04:55:52.683583 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:52.682851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:59957)
0806 04:55:52.683343 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:52.683419 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":277,"threads_started":1}
I20260806 04:55:53.106552 31383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:53.105910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:48573)
0806 04:55:53.106317 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:53.106391 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":208,"threads_started":1}
I20260806 04:55:53.528548 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:53.527879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:60731)
0806 04:55:53.528300 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:53.528383 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":219,"threads_started":1}
I20260806 04:55:53.685410 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:53.684716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:45005)
0806 04:55:53.685160 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:53.685245 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":271,"threads_started":1}
I20260806 04:55:54.108582 31386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:54.107686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:40045)
0806 04:55:54.108204 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:54.108338 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":276,"threads_started":1}
I20260806 04:55:54.521342 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 30892
I20260806 04:55:54.530334 31388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:54.529519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.131:44643)
0806 04:55:54.530042 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:54.530160 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":264,"threads_started":1}
I20260806 04:55:54.687044 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:54.686355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.129:42573)
0806 04:55:54.686798 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:54.686909 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":209,"threads_started":1}
I20260806 04:55:55.110277 31390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:55.109538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:41047 (local address 127.13.148.130:41999)
0806 04:55:55.110053 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:55.110135 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":270,"threads_started":1}
I20260806 04:55:55.113157 31387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:54.523209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.13.148.129:36149)
0806 04:55:54.525351 (+ 2142us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:54.525377 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:55.097590 (+572213us) server_negotiation.cc:407] Connection header received
0806 04:55:55.099281 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:55.099291 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:55.099743 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:55.100054 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:55.101587 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:55.102865 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:55.104056 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:55.104713 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:55.104918 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:55.107480 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:55.107487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:55.107502 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:55.107536 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:55.110384 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:55.111051 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:55.111057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:55.111063 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:55.111202 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:55.111499 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:55.111502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:55.111503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:55.111970 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:55.112180 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:55.112487 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 04:55:55.112904 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1903,"thread_start_us":1750,"threads_started":1}
I20260806 04:55:55.114506 31059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:55:55.190838 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 30892
I20260806 04:55:55.209048 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 31069
I20260806 04:55:55.225564 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:55.210139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42150 (local address 127.13.148.130:46439)
0806 04:55:55.210711 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:55.210724 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:55.210764 (+ 40us) server_negotiation.cc:407] Connection header received
0806 04:55:55.212283 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:55.212291 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:55.212706 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:55.213062 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:55.214262 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:55.215319 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:55.216379 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:55.217137 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:55.217278 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:55.219731 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:55.219738 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:55.219751 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:55.219788 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:55.222712 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:55.223302 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:55.223308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:55.223314 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:55.223522 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:55.223833 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:55.223836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:55.223838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:55.224279 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:55.224475 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:55.224902 (+ 427us) server_negotiation.cc:299] Negotiation successful
0806 04:55:55.225298 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":254,"threads_started":1}
I20260806 04:55:55.226756 31196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:55:55.303232 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 31069
I20260806 04:55:55.321967 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 31204
I20260806 04:55:55.337817 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:55.322950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.13.148.131:44299)
0806 04:55:55.323533 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:55.323547 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:55.323583 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:55:55.325006 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:55.325016 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:55.325410 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:55.325681 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:55.327059 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:55.328315 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:55.329560 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:55.330195 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:55.330322 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:55.332720 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:55.332727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:55.332738 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:55.332767 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:55.335145 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:55.335695 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:55.335701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:55.335705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:55.335845 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:55.336149 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:55.336151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:55.336153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:55.336556 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:55.336887 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:55.337252 (+ 365us) server_negotiation.cc:299] Negotiation successful
0806 04:55:55.337545 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":260,"threads_started":1}
I20260806 04:55:55.339180 31331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:55:55.413414 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 31204
2026-08-06T04:55:55Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64197 ms)
[----------] 30 tests from SecurityITest (282391 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:55:55 dist-test-slave-1x4g krb5kdc[31400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:55:55 dist-test-slave-1x4g krb5kdc[31400](info): set up 2 sockets
Aug 06 04:55:55 dist-test-slave-1x4g krb5kdc[31400](info): commencing operation
krb5kdc: starting...
W20260806 04:55:56.429672 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:55:56 dist-test-slave-1x4g krb5kdc[31400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:55:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:55:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:55:56.660429 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:39955
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:39955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:55:56.963547 31413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:55:56.963786 31413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:55:56.963861 31413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:55:56.969929 31413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:55:56.970016 31413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:55:56.970060 31413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:55:56.970108 31413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:55:56.970139 31413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:55:56.978295 31413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:39955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:39955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:55:56.979360 31413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:55:56.980480 31413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:55:56.991847 31418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:56.991884 31421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:56.991784 31419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:55:56.992318 31413 server_base.cc:1034] running on GCE node
I20260806 04:55:56.993101 31413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:55:56.994257 31413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:55:56.995481 31413 hybrid_clock.cc:648] HybridClock initialized: now 1785992156995448 us; error 37 us; skew 500 ppm
I20260806 04:55:56.995787 31413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:55:56 dist-test-slave-1x4g krb5kdc[31400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992156, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:55:56.998798 31413 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:55:57.000180 31413 webserver.cc:460] Webserver started at http://127.13.148.190:46567/ using document root <none> and password file <none>
I20260806 04:55:57.000811 31413 fs_manager.cc:359] Metadata directory not provided
I20260806 04:55:57.000942 31413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:55:57.001219 31413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:55:57.003847 31413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "947805bb8fe345c8a2d07feeb0438abc"
format_stamp: "Formatted at 2026-08-06 04:55:56 on dist-test-slave-1x4g"
server_key: "c65525b1652bfbbe9ea3d2d095450b20"
server_key_iv: "26d29c340fe36da1433d438a7f4cbe74"
server_key_version: "encryptionkey@0"
I20260806 04:55:57.004531 31413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "947805bb8fe345c8a2d07feeb0438abc"
format_stamp: "Formatted at 2026-08-06 04:55:56 on dist-test-slave-1x4g"
server_key: "c65525b1652bfbbe9ea3d2d095450b20"
server_key_iv: "26d29c340fe36da1433d438a7f4cbe74"
server_key_version: "encryptionkey@0"
I20260806 04:55:57.009122 31413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260806 04:55:57.012183 31428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:57.013326 31413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:55:57.013506 31413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "947805bb8fe345c8a2d07feeb0438abc"
format_stamp: "Formatted at 2026-08-06 04:55:56 on dist-test-slave-1x4g"
server_key: "c65525b1652bfbbe9ea3d2d095450b20"
server_key_iv: "26d29c340fe36da1433d438a7f4cbe74"
server_key_version: "encryptionkey@0"
I20260806 04:55:57.013689 31413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:55:57.029518 31413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:55:57.033522 31413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:55:57.033797 31413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:55:57.051375 31413 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:39955
I20260806 04:55:57.051396 31481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:39955 every 8 connection(s)
I20260806 04:55:57.052639 31413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:55:57.055995 31482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:55:57.061206 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 31413
I20260806 04:55:57.061414 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:55:57.061981 13906 external_mini_cluster.cc:1428] Setting key ec7f0f9b4f01d194b489f8fabf6f210a
I20260806 04:55:57.065984 31482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc: Bootstrap starting.
I20260806 04:55:57.069572 31482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc: Neither blocks nor log segments found. Creating new log.
I20260806 04:55:57.070560 31482 log.cc:824] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc: Log is configured to *not* fsync() on all Append() calls
I20260806 04:55:57.073489 31482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc: No bootstrap required, opened a new log
Aug 06 04:55:57 dist-test-slave-1x4g krb5kdc[31400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:55:57.081465 31482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947805bb8fe345c8a2d07feeb0438abc" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 39955 } }
I20260806 04:55:57.081809 31482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:55:57.081950 31482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 947805bb8fe345c8a2d07feeb0438abc, State: Initialized, Role: FOLLOWER
I20260806 04:55:57.082502 31482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947805bb8fe345c8a2d07feeb0438abc" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 39955 } }
I20260806 04:55:57.082734 31482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:55:57.082880 31482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:55:57.083083 31482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:55:57.084199 31485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:57.063891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.13.148.190:39955)
0806 04:55:57.064686 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:57.064696 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:57.064732 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:55:57.066351 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:57.066372 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:57.066694 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:57.067020 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:57.069085 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:57.070296 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:57.071903 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:57.072569 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:57.072695 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:57.077290 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:57.077297 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:57.077311 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:57.077344 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:57.080313 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:57.081008 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:57.081019 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:57.081025 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:57.081180 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:57.081552 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:57.081555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:57.081557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:57.082174 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:57.082469 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:57.082854 (+ 385us) server_negotiation.cc:299] Negotiation successful
0806 04:55:57.083282 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":367,"threads_started":1}
I20260806 04:55:57.085338 31482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947805bb8fe345c8a2d07feeb0438abc" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 39955 } }
I20260806 04:55:57.085781 31482 leader_election.cc:302] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 947805bb8fe345c8a2d07feeb0438abc; no voters:
I20260806 04:55:57.086577 31482 leader_election.cc:288] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:55:57.086681 31487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:55:57.087710 31487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [term 1 LEADER]: Becoming Leader. State: Replica: 947805bb8fe345c8a2d07feeb0438abc, State: Running, Role: LEADER
I20260806 04:55:57.088294 31487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947805bb8fe345c8a2d07feeb0438abc" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 39955 } }
I20260806 04:55:57.089175 31482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:55:57.091074 31489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 947805bb8fe345c8a2d07feeb0438abc. Latest consensus state: current_term: 1 leader_uuid: "947805bb8fe345c8a2d07feeb0438abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947805bb8fe345c8a2d07feeb0438abc" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 39955 } } }
I20260806 04:55:57.091132 31488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "947805bb8fe345c8a2d07feeb0438abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947805bb8fe345c8a2d07feeb0438abc" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 39955 } } }
I20260806 04:55:57.091471 31489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [sys.catalog]: This master's current role is: LEADER
I20260806 04:55:57.091496 31488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc [sys.catalog]: This master's current role is: LEADER
I20260806 04:55:57.094051 31493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:55:57.098088 31493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:55:57.106271 31493 catalog_manager.cc:1347] Generated new cluster ID: 31aa54adf7ee45fcaebe120967106ed5
I20260806 04:55:57.106371 31493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:55:57.116504 31493 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:55:57.117893 31493 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:55:57.126255 31493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc: Generated new TSK 0
I20260806 04:55:57.126957 31493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:55:57.188304 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:39955
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:55:57.503886 31508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:55:57.504158 31508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:55:57.504316 31508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:55:57.511300 31508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:55:57.511475 31508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:55:57.511660 31508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:55:57.521328 31508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:39955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:55:57.522610 31508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:55:57.523903 31508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:55:57.536305 31514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:57.536348 31513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:57.536378 31516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:55:57.537437 31508 server_base.cc:1034] running on GCE node
I20260806 04:55:57.538044 31508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:55:57.538990 31508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:55:57.540225 31508 hybrid_clock.cc:648] HybridClock initialized: now 1785992157540179 us; error 50 us; skew 500 ppm
I20260806 04:55:57.540532 31508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:55:57 dist-test-slave-1x4g krb5kdc[31400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992157, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:55:57.543531 31508 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:55:57.544919 31508 webserver.cc:460] Webserver started at http://127.13.148.129:46223/ using document root <none> and password file <none>
I20260806 04:55:57.545560 31508 fs_manager.cc:359] Metadata directory not provided
I20260806 04:55:57.545717 31508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:55:57.546025 31508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:55:57.548913 31508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "81b77f9caba344c0a006904843f95bd2"
format_stamp: "Formatted at 2026-08-06 04:55:57 on dist-test-slave-1x4g"
server_key: "b3040aef86f1a7c23acfed3cecc90eb0"
server_key_iv: "52a3853ca4ffbb3a704538eaa02d0b41"
server_key_version: "encryptionkey@0"
I20260806 04:55:57.549649 31508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "81b77f9caba344c0a006904843f95bd2"
format_stamp: "Formatted at 2026-08-06 04:55:57 on dist-test-slave-1x4g"
server_key: "b3040aef86f1a7c23acfed3cecc90eb0"
server_key_iv: "52a3853ca4ffbb3a704538eaa02d0b41"
server_key_version: "encryptionkey@0"
I20260806 04:55:57.554656 31508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 04:55:57.557938 31523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:57.559147 31508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:55:57.559299 31508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "81b77f9caba344c0a006904843f95bd2"
format_stamp: "Formatted at 2026-08-06 04:55:57 on dist-test-slave-1x4g"
server_key: "b3040aef86f1a7c23acfed3cecc90eb0"
server_key_iv: "52a3853ca4ffbb3a704538eaa02d0b41"
server_key_version: "encryptionkey@0"
I20260806 04:55:57.559553 31508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:55:57.574518 31508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:55:57.578262 31508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:55:57.578558 31508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:55:57.579402 31508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:55:57.580626 31508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:55:57.580725 31508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:57.580878 31508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:55:57.580921 31508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:57.611357 31508 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:45037
I20260806 04:55:57.611451 31638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:45037 every 8 connection(s)
I20260806 04:55:57.612514 31508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:55:57.613736 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 31508
I20260806 04:55:57.613930 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:55:57.614508 13906 external_mini_cluster.cc:1428] Setting key 992e20c5acdb8de810e5c716c6e3249a
Aug 06 04:55:57 dist-test-slave-1x4g krb5kdc[31400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992157, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:55:57.633778 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:57.614930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:49457 (local address 127.13.148.190:39955)
0806 04:55:57.617318 (+ 2388us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:57.617325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:57.617343 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:55:57.618820 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:57.618825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:57.618907 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:57.619016 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:57.620737 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:57.621818 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:57.623161 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:57.623786 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:57.623908 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:57.627585 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:57.627594 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:57.627601 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:57.627643 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:57.630841 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:57.631512 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:57.631519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:57.631524 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:57.631663 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:57.632022 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:57.632028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:57.632034 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:57.632364 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:57.632510 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:57.633116 (+ 606us) server_negotiation.cc:299] Negotiation successful
0806 04:55:57.633348 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2248,"thread_start_us":2156,"threads_started":1}
I20260806 04:55:57.634294 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:57.615581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:39955 (local address 127.13.148.129:49457)
0806 04:55:57.616342 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:57.616382 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 04:55:57.618542 (+ 2160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:55:57.619218 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:55:57.619228 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:55:57.619959 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:55:57.620574 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:57.620591 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:57.621953 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:57.621956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:57.623021 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:57.623031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:57.623979 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:57.623983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:57.624149 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:57.624754 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:55:57.624815 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:55:57.627352 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:55:57.630957 (+ 3605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:55:57.630963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:55:57.630973 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:55:57.631411 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:55:57.631767 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:55:57.631769 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:55:57.631771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:55:57.631925 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:55:57.632471 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:55:57.632476 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:55:57.632739 (+ 263us) client_negotiation.cc:769] Sending connection context
0806 04:55:57.633007 (+ 268us) client_negotiation.cc:240] Negotiation successful
0806 04:55:57.633315 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":318,"threads_started":1}
I20260806 04:55:57.635754 31639 heartbeater.cc:347] Connected to a master server at 127.13.148.190:39955
I20260806 04:55:57.636027 31639 heartbeater.cc:464] Registering TS with master...
I20260806 04:55:57.636947 31639 heartbeater.cc:511] Master 127.13.148.190:39955 requested a full tablet report, sending...
I20260806 04:55:57.639336 31446 ts_manager.cc:194] Registered new tserver with Master: 81b77f9caba344c0a006904843f95bd2 (127.13.148.129:45037)
I20260806 04:55:57.641304 31446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:49457
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:55:57.677289 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:39955
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:55:57.972626 31646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:55:57.972882 31646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:55:57.972965 31646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:55:57.979453 31646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:55:57.979538 31646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:55:57.979630 31646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:55:57.988646 31646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:39955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:55:57.989902 31646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:55:57.991178 31646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:55:58.003266 31652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:58.003357 31654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:58.003682 31651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:55:58.003919 31646 server_base.cc:1034] running on GCE node
I20260806 04:55:58.004427 31646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:55:58.005323 31646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:55:58.006600 31646 hybrid_clock.cc:648] HybridClock initialized: now 1785992158006548 us; error 54 us; skew 500 ppm
I20260806 04:55:58.006968 31646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:55:58 dist-test-slave-1x4g krb5kdc[31400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992158, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:55:58.010180 31646 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:55:58.011363 31646 webserver.cc:460] Webserver started at http://127.13.148.130:40091/ using document root <none> and password file <none>
I20260806 04:55:58.011960 31646 fs_manager.cc:359] Metadata directory not provided
I20260806 04:55:58.012095 31646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:55:58.012370 31646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:55:58.015034 31646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "2ecc2560da7e4d5a9c6cfb1fc46847a0"
format_stamp: "Formatted at 2026-08-06 04:55:58 on dist-test-slave-1x4g"
server_key: "453128498fda01d9ee9547bd740dd3ee"
server_key_iv: "5274a185c81ab68878d0b921011773c9"
server_key_version: "encryptionkey@0"
I20260806 04:55:58.015736 31646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "2ecc2560da7e4d5a9c6cfb1fc46847a0"
format_stamp: "Formatted at 2026-08-06 04:55:58 on dist-test-slave-1x4g"
server_key: "453128498fda01d9ee9547bd740dd3ee"
server_key_iv: "5274a185c81ab68878d0b921011773c9"
server_key_version: "encryptionkey@0"
I20260806 04:55:58.020762 31646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 04:55:58.023831 31661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:58.025043 31646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:55:58.025202 31646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "2ecc2560da7e4d5a9c6cfb1fc46847a0"
format_stamp: "Formatted at 2026-08-06 04:55:58 on dist-test-slave-1x4g"
server_key: "453128498fda01d9ee9547bd740dd3ee"
server_key_iv: "5274a185c81ab68878d0b921011773c9"
server_key_version: "encryptionkey@0"
I20260806 04:55:58.025369 31646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:55:58.038781 31646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:55:58.042676 31646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:55:58.042932 31646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:55:58.043730 31646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:55:58.044862 31646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:55:58.044955 31646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:58.045084 31646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:55:58.045148 31646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:58.076707 31646 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:44943
I20260806 04:55:58.076761 31780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:44943 every 8 connection(s)
I20260806 04:55:58.078054 31646 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:55:58.088308 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 31646
I20260806 04:55:58.088490 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:55:58.089006 13906 external_mini_cluster.cc:1428] Setting key 6f1b0263a5f02bf3c4bf6d975e27f9c4
Aug 06 04:55:58 dist-test-slave-1x4g krb5kdc[31400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992158, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:55:58.095919 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:58.080394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:34923 (local address 127.13.148.190:39955)
0806 04:55:58.080601 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:58.080605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:58.081692 (+ 1087us) server_negotiation.cc:407] Connection header received
0806 04:55:58.083275 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:58.083279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:58.083345 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:58.083432 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:58.084757 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.085484 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.086756 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.087082 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.087161 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:58.090661 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:58.090665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:58.090669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:58.090697 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:58.093330 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:58.093958 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:58.093963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:58.093971 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:58.094099 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:58.094480 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:58.094484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:58.094487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:58.094713 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:58.094870 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:58.095425 (+ 555us) server_negotiation.cc:299] Negotiation successful
0806 04:55:58.095623 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 04:55:58.096769 31784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:58.080872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:39955 (local address 127.13.148.130:34923)
0806 04:55:58.081498 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:58.081573 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0806 04:55:58.083032 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:55:58.083625 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:55:58.083632 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:55:58.084120 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:55:58.084617 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:58.084632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.085611 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.085614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:58.086648 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:58.086658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.087236 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.087240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:58.087431 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:58.088019 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:55:58.088040 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:55:58.090472 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:55:58.093459 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:55:58.093466 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:55:58.093477 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:55:58.093840 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:55:58.094201 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:55:58.094205 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:55:58.094208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:55:58.094377 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:55:58.094831 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:55:58.094838 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:55:58.095127 (+ 289us) client_negotiation.cc:769] Sending connection context
0806 04:55:58.095356 (+ 229us) client_negotiation.cc:240] Negotiation successful
0806 04:55:58.095782 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":247,"threads_started":1}
I20260806 04:55:58.098263 31781 heartbeater.cc:347] Connected to a master server at 127.13.148.190:39955
I20260806 04:55:58.098569 31781 heartbeater.cc:464] Registering TS with master...
I20260806 04:55:58.099339 31781 heartbeater.cc:511] Master 127.13.148.190:39955 requested a full tablet report, sending...
I20260806 04:55:58.100764 31446 ts_manager.cc:194] Registered new tserver with Master: 2ecc2560da7e4d5a9c6cfb1fc46847a0 (127.13.148.130:44943)
I20260806 04:55:58.101593 31446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:34923
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
I20260806 04:55:58.143959 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:39955
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:55:58.439791 31787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:55:58.440028 31787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:55:58.440111 31787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:55:58.446314 31787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:55:58.446401 31787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:55:58.446537 31787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:55:58.454705 31787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:39955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:55:58.455771 31787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:55:58.457006 31787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:55:58.468930 31795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:58.469018 31793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:55:58.469151 31792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:55:58.469713 31787 server_base.cc:1034] running on GCE node
I20260806 04:55:58.470161 31787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:55:58.470867 31787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:55:58.472075 31787 hybrid_clock.cc:648] HybridClock initialized: now 1785992158472018 us; error 47 us; skew 500 ppm
I20260806 04:55:58.472373 31787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:55:58 dist-test-slave-1x4g krb5kdc[31400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992158, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:55:58.475063 31787 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:55:58.476277 31787 webserver.cc:460] Webserver started at http://127.13.148.131:33615/ using document root <none> and password file <none>
I20260806 04:55:58.477006 31787 fs_manager.cc:359] Metadata directory not provided
I20260806 04:55:58.477159 31787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:55:58.477470 31787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:55:58.479982 31787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "f0b82359fc2f44cf8d22368bf178ff92"
format_stamp: "Formatted at 2026-08-06 04:55:58 on dist-test-slave-1x4g"
server_key: "6fe6ca6dfeb4b30155d5abb510cc502c"
server_key_iv: "dcd5c149996bafaeeb44b57b23fbc4b7"
server_key_version: "encryptionkey@0"
I20260806 04:55:58.480659 31787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "f0b82359fc2f44cf8d22368bf178ff92"
format_stamp: "Formatted at 2026-08-06 04:55:58 on dist-test-slave-1x4g"
server_key: "6fe6ca6dfeb4b30155d5abb510cc502c"
server_key_iv: "dcd5c149996bafaeeb44b57b23fbc4b7"
server_key_version: "encryptionkey@0"
I20260806 04:55:58.485337 31787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 04:55:58.488416 31802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:58.489660 31787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:55:58.489846 31787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "f0b82359fc2f44cf8d22368bf178ff92"
format_stamp: "Formatted at 2026-08-06 04:55:58 on dist-test-slave-1x4g"
server_key: "6fe6ca6dfeb4b30155d5abb510cc502c"
server_key_iv: "dcd5c149996bafaeeb44b57b23fbc4b7"
server_key_version: "encryptionkey@0"
I20260806 04:55:58.490032 31787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:55:58.505733 31787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:55:58.509243 31787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:55:58.509538 31787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:55:58.510360 31787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:55:58.511615 31787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:55:58.511749 31787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:58.511894 31787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:55:58.511963 31787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:55:58.542896 31787 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:32839
I20260806 04:55:58.542944 31918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:32839 every 8 connection(s)
I20260806 04:55:58.544121 31787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:55:58.554265 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 31787
I20260806 04:55:58.554443 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:55:58.554852 13906 external_mini_cluster.cc:1428] Setting key 45cce047d49e992b7fff819f3ae67a06
Aug 06 04:55:58 dist-test-slave-1x4g krb5kdc[31400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992158, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:55:58.562501 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:58.546455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:33751 (local address 127.13.148.190:39955)
0806 04:55:58.546612 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:58.546616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:58.547651 (+ 1035us) server_negotiation.cc:407] Connection header received
0806 04:55:58.549431 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:58.549436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:58.549522 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:58.549622 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:58.550950 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.551626 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.553133 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.553525 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.553646 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:58.556947 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:58.556953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:58.556958 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:58.556996 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:58.559751 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:58.560395 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:58.560400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:58.560407 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:58.560514 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:58.560945 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:58.560949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:58.560952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:58.561197 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:58.561343 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:58.561941 (+ 598us) server_negotiation.cc:299] Negotiation successful
0806 04:55:58.562181 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:55:58.563321 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:58.546830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:39955 (local address 127.13.148.131:33751)
0806 04:55:58.547488 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:58.547536 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:55:58.549188 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:55:58.549831 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:55:58.549837 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:55:58.550292 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:55:58.550791 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:58.550807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.551787 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.551791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:58.553003 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:58.553019 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.553724 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.553732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:58.553992 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:58.554634 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:55:58.554658 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:55:58.556738 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:55:58.559895 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:55:58.559903 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:55:58.559915 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:55:58.560277 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:55:58.560614 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:55:58.560627 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:55:58.560630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:55:58.560836 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:55:58.561314 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:55:58.561320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:55:58.561615 (+ 295us) client_negotiation.cc:769] Sending connection context
0806 04:55:58.561839 (+ 224us) client_negotiation.cc:240] Negotiation successful
0806 04:55:58.562245 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":242,"threads_started":1}
I20260806 04:55:58.564461 31919 heartbeater.cc:347] Connected to a master server at 127.13.148.190:39955
I20260806 04:55:58.564710 31919 heartbeater.cc:464] Registering TS with master...
I20260806 04:55:58.565323 31919 heartbeater.cc:511] Master 127.13.148.190:39955 requested a full tablet report, sending...
I20260806 04:55:58.566525 31446 ts_manager.cc:194] Registered new tserver with Master: f0b82359fc2f44cf8d22368bf178ff92 (127.13.148.131:32839)
I20260806 04:55:58.567212 31446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:33751
I20260806 04:55:58.569885 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:55:58.585345 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:58.573946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51292 (local address 127.13.148.190:39955)
0806 04:55:58.574092 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:58.574096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:58.574497 (+ 401us) server_negotiation.cc:407] Connection header received
0806 04:55:58.574680 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:58.574684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:58.574733 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:58.574819 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:58.576034 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.576716 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.578046 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.578457 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.578614 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:58.580717 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:58.580721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:58.580724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:58.580748 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:58.582980 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:58.583603 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:58.583607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:58.583614 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:58.583729 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:58.584065 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:58.584069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:58.584072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:58.584325 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:58.584554 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:58.584898 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 04:55:58.585099 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:55:58.588827 31446 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:55:58.595434 31446 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:55:58.616534 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:58.606941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.13.148.129:45037)
0806 04:55:58.607656 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:58.607663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:58.607842 (+ 179us) server_negotiation.cc:407] Connection header received
0806 04:55:58.608144 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:58.608152 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:58.608419 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:58.608653 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:55:58.609614 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.611227 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.613625 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.615053 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.615245 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:58.615427 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:58.616061 (+ 634us) server_negotiation.cc:299] Negotiation successful
0806 04:55:58.616203 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":446,"threads_started":1}
I20260806 04:55:58.617553 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:58.606724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:45037 (local address 127.0.0.1:57558)
0806 04:55:58.607709 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:58.607741 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 04:55:58.607917 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:55:58.608939 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:55:58.608959 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:55:58.608996 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:55:58.609408 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:58.609419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.611384 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.611388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:58.613429 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:58.613450 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.615595 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.615601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:58.615788 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:58.615839 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 04:55:58.616878 (+ 1039us) client_negotiation.cc:240] Negotiation successful
0806 04:55:58.617087 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":1,"thread_start_us":615,"threads_started":1}
I20260806 04:55:58.618165 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:58.608726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:44943 (local address 127.0.0.1:46608)
0806 04:55:58.609261 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:58.609280 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:55:58.609392 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:55:58.610806 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:55:58.610816 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:55:58.610846 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:55:58.611306 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:58.611317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.613296 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.613300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:58.615319 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:58.615337 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.617252 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.617259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:58.617465 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:58.617508 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 04:55:58.617591 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 04:55:58.617736 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":4,"thread_start_us":768,"threads_started":1}
I20260806 04:55:58.618353 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:58.608386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.13.148.131:32839)
0806 04:55:58.609997 (+ 1611us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:58.610006 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:58.610037 (+ 31us) server_negotiation.cc:407] Connection header received
0806 04:55:58.610127 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:58.610135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:58.610331 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:58.610508 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:55:58.611393 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.612813 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.615347 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.616880 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.617061 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:58.617219 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:58.617880 (+ 661us) server_negotiation.cc:299] Negotiation successful
0806 04:55:58.618041 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1422,"thread_start_us":1259,"threads_started":1}
I20260806 04:55:58.618345 31933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:58.608094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:32839 (local address 127.0.0.1:43090)
0806 04:55:58.608630 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:58.608646 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:55:58.608747 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:55:58.610745 (+ 1998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:55:58.610749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:55:58.610771 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:55:58.611176 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:58.611193 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.612963 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.612967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:58.615113 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:58.615131 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.617373 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.617379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:58.617681 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:58.617719 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 04:55:58.617804 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 04:55:58.617930 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":166,"thread_start_us":253,"threads_started":1}
I20260806 04:55:58.619426 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:58.609282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46608 (local address 127.13.148.130:44943)
0806 04:55:58.610238 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:58.610246 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:58.610271 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:55:58.610367 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:58.610376 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:58.610596 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:58.611522 (+ 926us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:55:58.611728 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.613142 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.615496 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:58.617011 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:58.618780 (+ 1769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:58.618935 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:58.619005 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 04:55:58.619151 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":869,"threads_started":1}
I20260806 04:55:58.621691 31573 tablet_service.cc:1467] Processing CreateTablet for tablet 7b2a77a79de148eea725d506be61ed4c (DEFAULT_TABLE table=test-table [id=5274b6259a4349f980b4ba63f8065dbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:55:58.622071 31853 tablet_service.cc:1467] Processing CreateTablet for tablet 7b2a77a79de148eea725d506be61ed4c (DEFAULT_TABLE table=test-table [id=5274b6259a4349f980b4ba63f8065dbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:55:58.622581 31713 tablet_service.cc:1467] Processing CreateTablet for tablet 7b2a77a79de148eea725d506be61ed4c (DEFAULT_TABLE table=test-table [id=5274b6259a4349f980b4ba63f8065dbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:55:58.623137 31573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2a77a79de148eea725d506be61ed4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:55:58.623345 31853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2a77a79de148eea725d506be61ed4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:55:58.623704 31713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2a77a79de148eea725d506be61ed4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:55:58.634124 31938 tablet_bootstrap.cc:492] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0: Bootstrap starting.
I20260806 04:55:58.636413 31940 tablet_bootstrap.cc:492] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92: Bootstrap starting.
I20260806 04:55:58.638018 31939 tablet_bootstrap.cc:492] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2: Bootstrap starting.
I20260806 04:55:58.638765 31938 tablet_bootstrap.cc:654] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0: Neither blocks nor log segments found. Creating new log.
I20260806 04:55:58.639885 31940 tablet_bootstrap.cc:654] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92: Neither blocks nor log segments found. Creating new log.
I20260806 04:55:58.640035 31938 log.cc:824] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0: Log is configured to *not* fsync() on all Append() calls
I20260806 04:55:58.640962 31939 tablet_bootstrap.cc:654] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2: Neither blocks nor log segments found. Creating new log.
I20260806 04:55:58.641075 31940 log.cc:824] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92: Log is configured to *not* fsync() on all Append() calls
I20260806 04:55:58.642125 31939 log.cc:824] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2: Log is configured to *not* fsync() on all Append() calls
I20260806 04:55:58.643390 31938 tablet_bootstrap.cc:492] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0: No bootstrap required, opened a new log
I20260806 04:55:58.643641 31938 ts_tablet_manager.cc:1397] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 04:55:58.644079 31940 tablet_bootstrap.cc:492] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92: No bootstrap required, opened a new log
I20260806 04:55:58.644362 31940 ts_tablet_manager.cc:1397] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260806 04:55:58.644907 31639 heartbeater.cc:503] Master 127.13.148.190:39955 was elected leader, sending a full tablet report...
I20260806 04:55:58.651046 31939 tablet_bootstrap.cc:492] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2: No bootstrap required, opened a new log
I20260806 04:55:58.651327 31939 ts_tablet_manager.cc:1397] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.012s
I20260806 04:55:58.653373 31938 raft_consensus.cc:348] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b77f9caba344c0a006904843f95bd2" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 45037 } } peers { permanent_uuid: "f0b82359fc2f44cf8d22368bf178ff92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 32839 } } peers { permanent_uuid: "2ecc2560da7e4d5a9c6cfb1fc46847a0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44943 } }
I20260806 04:55:58.653744 31938 raft_consensus.cc:374] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:55:58.653855 31938 raft_consensus.cc:729] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecc2560da7e4d5a9c6cfb1fc46847a0, State: Initialized, Role: FOLLOWER
I20260806 04:55:58.654465 31938 consensus_queue.cc:260] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b77f9caba344c0a006904843f95bd2" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 45037 } } peers { permanent_uuid: "f0b82359fc2f44cf8d22368bf178ff92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 32839 } } peers { permanent_uuid: "2ecc2560da7e4d5a9c6cfb1fc46847a0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44943 } }
I20260806 04:55:58.654814 31940 raft_consensus.cc:348] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b77f9caba344c0a006904843f95bd2" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 45037 } } peers { permanent_uuid: "f0b82359fc2f44cf8d22368bf178ff92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 32839 } } peers { permanent_uuid: "2ecc2560da7e4d5a9c6cfb1fc46847a0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44943 } }
I20260806 04:55:58.655257 31940 raft_consensus.cc:374] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:55:58.655392 31940 raft_consensus.cc:729] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0b82359fc2f44cf8d22368bf178ff92, State: Initialized, Role: FOLLOWER
I20260806 04:55:58.655459 31781 heartbeater.cc:503] Master 127.13.148.190:39955 was elected leader, sending a full tablet report...
I20260806 04:55:58.656096 31938 ts_tablet_manager.cc:1428] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
I20260806 04:55:58.656005 31940 consensus_queue.cc:260] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b77f9caba344c0a006904843f95bd2" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 45037 } } peers { permanent_uuid: "f0b82359fc2f44cf8d22368bf178ff92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 32839 } } peers { permanent_uuid: "2ecc2560da7e4d5a9c6cfb1fc46847a0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44943 } }
I20260806 04:55:58.657862 31940 ts_tablet_manager.cc:1428] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92: Time spent starting tablet: real 0.013s user 0.005s sys 0.009s
I20260806 04:55:58.658052 31919 heartbeater.cc:503] Master 127.13.148.190:39955 was elected leader, sending a full tablet report...
I20260806 04:55:58.661268 31939 raft_consensus.cc:348] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b77f9caba344c0a006904843f95bd2" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 45037 } } peers { permanent_uuid: "f0b82359fc2f44cf8d22368bf178ff92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 32839 } } peers { permanent_uuid: "2ecc2560da7e4d5a9c6cfb1fc46847a0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44943 } }
I20260806 04:55:58.661679 31939 raft_consensus.cc:374] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:55:58.661791 31939 raft_consensus.cc:729] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b77f9caba344c0a006904843f95bd2, State: Initialized, Role: FOLLOWER
I20260806 04:55:58.662298 31939 consensus_queue.cc:260] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b77f9caba344c0a006904843f95bd2" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 45037 } } peers { permanent_uuid: "f0b82359fc2f44cf8d22368bf178ff92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 32839 } } peers { permanent_uuid: "2ecc2560da7e4d5a9c6cfb1fc46847a0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44943 } }
I20260806 04:55:58.663749 31939 ts_tablet_manager.cc:1428] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260806 04:55:58.795624 31920 tablet.cc:2369] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:55:58.830004 31783 tablet.cc:2369] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:55:58.864571 31640 tablet.cc:2369] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:55:59.034950 31945 raft_consensus.cc:482] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:55:59.035240 31945 raft_consensus.cc:504] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b77f9caba344c0a006904843f95bd2" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 45037 } } peers { permanent_uuid: "f0b82359fc2f44cf8d22368bf178ff92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 32839 } } peers { permanent_uuid: "2ecc2560da7e4d5a9c6cfb1fc46847a0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44943 } }
I20260806 04:55:59.036898 31945 leader_election.cc:288] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81b77f9caba344c0a006904843f95bd2 (127.13.148.129:45037), 2ecc2560da7e4d5a9c6cfb1fc46847a0 (127.13.148.130:44943)
I20260806 04:55:59.043246 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:59.037252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:45037 (local address 127.13.148.131:44807)
0806 04:55:59.037602 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:59.037622 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:55:59.037775 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:55:59.038747 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:55:59.038753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:55:59.038785 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:55:59.039022 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:59.039034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:59.040162 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:59.040168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:59.041500 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:59.041511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:59.042541 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:59.042546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:59.042736 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:59.042765 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 04:55:59.042846 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 04:55:59.042962 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":1}
I20260806 04:55:59.043262 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:59.037241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:44807 (local address 127.13.148.129:45037)
0806 04:55:59.037436 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:59.037441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:59.037715 (+ 274us) server_negotiation.cc:407] Connection header received
0806 04:55:59.038384 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:59.038392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:59.038505 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:59.038602 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:55:59.039204 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:59.040044 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:59.041651 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:59.042399 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:59.042508 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:59.042562 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:59.042898 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 04:55:59.043023 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:55:59.043342 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:59.037903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:55715 (local address 127.13.148.130:44943)
0806 04:55:59.038110 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:59.038116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:59.038414 (+ 298us) server_negotiation.cc:407] Connection header received
0806 04:55:59.038760 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:59.038766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:59.038880 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:59.038991 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:55:59.039549 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:59.040262 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:59.041850 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:59.042558 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:59.042697 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:59.042758 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:59.043009 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 04:55:59.043130 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 04:55:59.043423 31947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:59.037581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:44943 (local address 127.13.148.131:55715)
0806 04:55:59.038301 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0806 04:55:59.038322 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 04:55:59.038545 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:55:59.039040 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:55:59.039044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:55:59.039070 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:55:59.039397 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:59.039408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:59.040391 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:59.040398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:59.041710 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:55:59.041723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:55:59.042727 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:55:59.042730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:55:59.042855 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:59.042884 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 04:55:59.042964 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 04:55:59.043079 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":1,"thread_start_us":418,"threads_started":1}
I20260806 04:55:59.044348 31733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2a77a79de148eea725d506be61ed4c" candidate_uuid: "f0b82359fc2f44cf8d22368bf178ff92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecc2560da7e4d5a9c6cfb1fc46847a0" is_pre_election: true
I20260806 04:55:59.044346 31593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2a77a79de148eea725d506be61ed4c" candidate_uuid: "f0b82359fc2f44cf8d22368bf178ff92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b77f9caba344c0a006904843f95bd2" is_pre_election: true
I20260806 04:55:59.044942 31733 raft_consensus.cc:2454] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0b82359fc2f44cf8d22368bf178ff92 in term 0.
I20260806 04:55:59.044951 31593 raft_consensus.cc:2454] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0b82359fc2f44cf8d22368bf178ff92 in term 0.
I20260806 04:55:59.045601 31806 leader_election.cc:302] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecc2560da7e4d5a9c6cfb1fc46847a0, f0b82359fc2f44cf8d22368bf178ff92; no voters:
I20260806 04:55:59.045976 31945 raft_consensus.cc:2781] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:55:59.046119 31945 raft_consensus.cc:482] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:55:59.046240 31945 raft_consensus.cc:3037] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:55:59.048414 31945 raft_consensus.cc:504] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b77f9caba344c0a006904843f95bd2" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 45037 } } peers { permanent_uuid: "f0b82359fc2f44cf8d22368bf178ff92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 32839 } } peers { permanent_uuid: "2ecc2560da7e4d5a9c6cfb1fc46847a0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44943 } }
I20260806 04:55:59.049047 31945 leader_election.cc:288] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [CANDIDATE]: Term 1 election: Requested vote from peers 81b77f9caba344c0a006904843f95bd2 (127.13.148.129:45037), 2ecc2560da7e4d5a9c6cfb1fc46847a0 (127.13.148.130:44943)
I20260806 04:55:59.049551 31593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2a77a79de148eea725d506be61ed4c" candidate_uuid: "f0b82359fc2f44cf8d22368bf178ff92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b77f9caba344c0a006904843f95bd2"
I20260806 04:55:59.049618 31733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2a77a79de148eea725d506be61ed4c" candidate_uuid: "f0b82359fc2f44cf8d22368bf178ff92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecc2560da7e4d5a9c6cfb1fc46847a0"
I20260806 04:55:59.049741 31593 raft_consensus.cc:3037] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:55:59.049827 31733 raft_consensus.cc:3037] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:55:59.052047 31593 raft_consensus.cc:2454] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0b82359fc2f44cf8d22368bf178ff92 in term 1.
I20260806 04:55:59.052431 31806 leader_election.cc:302] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81b77f9caba344c0a006904843f95bd2, f0b82359fc2f44cf8d22368bf178ff92; no voters:
I20260806 04:55:59.052604 31733 raft_consensus.cc:2454] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0b82359fc2f44cf8d22368bf178ff92 in term 1.
I20260806 04:55:59.052714 31945 raft_consensus.cc:2781] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:55:59.061086 31945 raft_consensus.cc:686] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [term 1 LEADER]: Becoming Leader. State: Replica: f0b82359fc2f44cf8d22368bf178ff92, State: Running, Role: LEADER
I20260806 04:55:59.061618 31945 consensus_queue.cc:237] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b77f9caba344c0a006904843f95bd2" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 45037 } } peers { permanent_uuid: "f0b82359fc2f44cf8d22368bf178ff92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 32839 } } peers { permanent_uuid: "2ecc2560da7e4d5a9c6cfb1fc46847a0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44943 } }
I20260806 04:55:59.066417 31445 catalog_manager.cc:5495] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0b82359fc2f44cf8d22368bf178ff92 (127.13.148.131). New cstate: current_term: 1 leader_uuid: "f0b82359fc2f44cf8d22368bf178ff92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b77f9caba344c0a006904843f95bd2" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 45037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0b82359fc2f44cf8d22368bf178ff92" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 32839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ecc2560da7e4d5a9c6cfb1fc46847a0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44943 } health_report { overall_health: UNKNOWN } } }
I20260806 04:55:59.182777 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:59.177850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43092 (local address 127.13.148.131:32839)
0806 04:55:59.178299 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:59.178303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:59.178457 (+ 154us) server_negotiation.cc:407] Connection header received
0806 04:55:59.178639 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:59.178642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:59.178715 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:59.178842 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:55:59.179200 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:59.179789 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:59.181097 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:59.181434 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:59.181518 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:59.181893 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:55:59.182339 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:55:59.182433 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:59.182512 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 04:55:59.182567 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":244,"threads_started":1}
I20260806 04:55:59.189909 31733 raft_consensus.cc:1261] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0 [term 1 FOLLOWER]: Refusing update from remote peer f0b82359fc2f44cf8d22368bf178ff92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:55:59.189909 31593 raft_consensus.cc:1261] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2 [term 1 FOLLOWER]: Refusing update from remote peer f0b82359fc2f44cf8d22368bf178ff92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:55:59.190752 31948 consensus_queue.cc:1035] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecc2560da7e4d5a9c6cfb1fc46847a0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:55:59.191066 31945 consensus_queue.cc:1035] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b77f9caba344c0a006904843f95bd2" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 45037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:55:59.203344 31953 mvcc.cc:205] Tried to move back new op lower bound from 7315423884032278528 to 7315423883520327680. Current Snapshot: MvccSnapshot[applied={T|T < 7315423884032278528}]
I20260806 04:55:59.215260 31445 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:55:59.215489 31445 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:55:59.218928 31445 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 947805bb8fe345c8a2d07feeb0438abc: Sending DeleteTablet for 3 replicas of tablet 7b2a77a79de148eea725d506be61ed4c
I20260806 04:55:59.220104 31853 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b2a77a79de148eea725d506be61ed4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:55:59 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:43090
I20260806 04:55:59.220119 31573 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b2a77a79de148eea725d506be61ed4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:55:59 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:57558
I20260806 04:55:59.220160 31713 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b2a77a79de148eea725d506be61ed4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:55:59 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:46608
I20260806 04:55:59.220930 31966 tablet_replica.cc:330] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92: stopping tablet replica
I20260806 04:55:59.221360 31966 raft_consensus.cc:2229] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:55:59.221515 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 31508
I20260806 04:55:59.222159 31966 raft_consensus.cc:2258] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:55:59.222839 31965 tablet_replica.cc:330] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2: stopping tablet replica
I20260806 04:55:59.223186 31965 raft_consensus.cc:2229] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:55:59.223630 31965 raft_consensus.cc:2258] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:55:59.223881 31966 ts_tablet_manager.cc:1905] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:55:59.229130 31968 tablet_replica.cc:330] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0: stopping tablet replica
I20260806 04:55:59.229499 31968 raft_consensus.cc:2229] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:55:59.229970 31968 raft_consensus.cc:2258] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:55:59.230692 31966 ts_tablet_manager.cc:1918] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:55:59.230880 31966 log.cc:1196] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/7b2a77a79de148eea725d506be61ed4c
I20260806 04:55:59.231366 31965 ts_tablet_manager.cc:1905] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:55:59.231451 31966 ts_tablet_manager.cc:1939] T 7b2a77a79de148eea725d506be61ed4c P f0b82359fc2f44cf8d22368bf178ff92: Deleting consensus metadata
I20260806 04:55:59.231678 31968 ts_tablet_manager.cc:1905] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:55:59.233058 31432 catalog_manager.cc:4841] TS f0b82359fc2f44cf8d22368bf178ff92 (127.13.148.131:32839): tablet 7b2a77a79de148eea725d506be61ed4c (table test-table [id=5274b6259a4349f980b4ba63f8065dbb]) successfully deleted
I20260806 04:55:59.236982 31965 ts_tablet_manager.cc:1918] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:55:59.237156 31965 log.cc:1196] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/7b2a77a79de148eea725d506be61ed4c
I20260806 04:55:59.237596 31965 ts_tablet_manager.cc:1939] T 7b2a77a79de148eea725d506be61ed4c P 81b77f9caba344c0a006904843f95bd2: Deleting consensus metadata
I20260806 04:55:59.238493 31968 ts_tablet_manager.cc:1918] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:55:59.238694 31968 log.cc:1196] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/7b2a77a79de148eea725d506be61ed4c
I20260806 04:55:59.239241 31968 ts_tablet_manager.cc:1939] T 7b2a77a79de148eea725d506be61ed4c P 2ecc2560da7e4d5a9c6cfb1fc46847a0: Deleting consensus metadata
I20260806 04:55:59.239274 31432 catalog_manager.cc:4841] TS 81b77f9caba344c0a006904843f95bd2 (127.13.148.129:45037): tablet 7b2a77a79de148eea725d506be61ed4c (table test-table [id=5274b6259a4349f980b4ba63f8065dbb]) successfully deleted
I20260806 04:55:59.240908 31432 catalog_manager.cc:4841] TS 2ecc2560da7e4d5a9c6cfb1fc46847a0 (127.13.148.130:44943): tablet 7b2a77a79de148eea725d506be61ed4c (table test-table [id=5274b6259a4349f980b4ba63f8065dbb]) successfully deleted
Aug 06 04:55:59 dist-test-slave-1x4g krb5kdc[31400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:55:59.249338 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:59.223458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57560 (local address 127.13.148.129:45037)
0806 04:55:59.223609 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:59.223619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:59.228502 (+ 4883us) server_negotiation.cc:407] Connection header received
0806 04:55:59.228700 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:59.228707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:59.228786 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:59.228932 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:59.230777 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:59.231784 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:59.233811 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:59.234481 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:59.234578 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:59.244101 (+ 9523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:59.244111 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:59.244121 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:59.244152 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:59.246527 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:59.247275 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:59.247281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:59.247282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:59.247421 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:59.247880 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:59.247883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:59.247885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:59.248229 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:59.248439 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:59.248935 (+ 496us) server_negotiation.cc:299] Negotiation successful
0806 04:55:59.249084 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:55:59.249868 31634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:55:59.330473 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 31508
I20260806 04:55:59.352751 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 31646
Aug 06 04:55:59 dist-test-slave-1x4g krb5kdc[31400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:55:59.369046 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:59.353641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.13.148.130:44943)
0806 04:55:59.353828 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:59.353834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:59.353857 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:55:59.353980 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:59.353986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:59.354080 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:59.354193 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:59.355347 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:59.356368 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:59.357733 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:59.358412 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:59.358558 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:59.362732 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:59.362751 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:59.362773 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:59.362816 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:59.366223 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:59.366810 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:59.366820 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:59.366824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:59.366990 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:59.367311 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:59.367316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:59.367320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:59.367834 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:59.368161 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:59.368330 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 04:55:59.368585 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 04:55:59.369706 31776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:55:59.451406 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 31646
I20260806 04:55:59.475435 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 31787
Aug 06 04:55:59 dist-test-slave-1x4g krb5kdc[31400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
I20260806 04:55:59.492918 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:55:59.476190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42758 (local address 127.13.148.131:32839)
0806 04:55:59.476401 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 04:55:59.476407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:55:59.476428 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:55:59.476525 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:55:59.476530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:55:59.476631 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:55:59.476761 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:55:59.478111 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:59.479247 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:59.480448 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:55:59.481094 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:55:59.481226 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:55:59.485563 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:55:59.485576 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:55:59.485595 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:55:59.485636 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:55:59.489990 (+ 4354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:59.490681 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:59.490690 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:59.490700 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:59.490823 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:55:59.491187 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:55:59.491192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:55:59.491195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:55:59.491582 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:55:59.491903 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0806 04:55:59.492184 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 04:55:59.492493 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 04:55:59.493656 31914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:55:59.604133 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 31787
I20260806 04:55:59.623649 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 31413
I20260806 04:55:59.624727 31477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:55:59.726159 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 31413
2026-08-06T04:55:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4319 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T04:55:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:55:59Z Disabled control of system clock
I20260806 04:55:59.805768 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:37885
--webserver_interface=127.13.148.190
--webserver_port=0
--builtin_ntp_servers=127.13.148.148:43729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:37885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:56:00.114991 31977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:00.115280 31977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:00.115356 31977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:00.122364 31977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:00.122493 31977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:00.122532 31977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:00.122598 31977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:00.122635 31977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:00.131194 31977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:43729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:37885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:37885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:00.132445 31977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:00.134151 31977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:00.145576 31982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:00.145509 31985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:00.145476 31983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:00.146749 31977 server_base.cc:1034] running on GCE node
I20260806 04:56:00.147424 31977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:00.148839 31977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:00.150066 31977 hybrid_clock.cc:648] HybridClock initialized: now 1785992160150023 us; error 48 us; skew 500 ppm
I20260806 04:56:00.150389 31977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:56:00.152381 31977 webserver.cc:460] Webserver started at http://127.13.148.190:40413/ using document root <none> and password file <none>
I20260806 04:56:00.153070 31977 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:00.153187 31977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:00.153482 31977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:00.156224 31977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "0be7b48776054b60bebbaa7347c3c5a6"
format_stamp: "Formatted at 2026-08-06 04:56:00 on dist-test-slave-1x4g"
server_key: "2e5940c6383ce43b08dded276e8ff445"
server_key_iv: "aca63183a0e5e44acf989e7ff66438eb"
server_key_version: "encryptionkey@0"
I20260806 04:56:00.157130 31977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "0be7b48776054b60bebbaa7347c3c5a6"
format_stamp: "Formatted at 2026-08-06 04:56:00 on dist-test-slave-1x4g"
server_key: "2e5940c6383ce43b08dded276e8ff445"
server_key_iv: "aca63183a0e5e44acf989e7ff66438eb"
server_key_version: "encryptionkey@0"
I20260806 04:56:00.161901 31977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260806 04:56:00.164997 31992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:00.166296 31977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 04:56:00.166425 31977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "0be7b48776054b60bebbaa7347c3c5a6"
format_stamp: "Formatted at 2026-08-06 04:56:00 on dist-test-slave-1x4g"
server_key: "2e5940c6383ce43b08dded276e8ff445"
server_key_iv: "aca63183a0e5e44acf989e7ff66438eb"
server_key_version: "encryptionkey@0"
I20260806 04:56:00.166575 31977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:00.189538 31977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:00.190295 31977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:00.190532 31977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:00.207342 31977 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:37885
I20260806 04:56:00.207367 32047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:37885 every 8 connection(s)
I20260806 04:56:00.208673 31977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:00.212348 32048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:00.216185 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 31977
I20260806 04:56:00.216351 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:00.216946 13906 external_mini_cluster.cc:1428] Setting key 04736aec1216ce1122f7c70d44a5de6f
I20260806 04:56:00.223076 32048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6: Bootstrap starting.
I20260806 04:56:00.226392 32048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:00.227648 32048 log.cc:824] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:00.230556 32051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:00.218843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.13.148.190:37885)
0806 04:56:00.219742 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:00.219752 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:00.219789 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:56:00.221660 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:00.221680 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:00.222078 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:00.222380 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:00.222959 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:00.224186 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:00.226010 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:00.226757 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:00.226904 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:00.228034 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:00.228041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:00.228054 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:00.228907 (+ 853us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:00.229009 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:00.229034 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:00.229139 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:00.229383 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 04:56:00.229683 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":341,"threads_started":1}
I20260806 04:56:00.230726 32048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6: No bootstrap required, opened a new log
I20260806 04:56:00.237911 32048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be7b48776054b60bebbaa7347c3c5a6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37885 } }
I20260806 04:56:00.238332 32048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:00.238497 32048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be7b48776054b60bebbaa7347c3c5a6, State: Initialized, Role: FOLLOWER
I20260806 04:56:00.239058 32048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be7b48776054b60bebbaa7347c3c5a6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37885 } }
I20260806 04:56:00.239297 32048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:00.239449 32048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:00.239620 32048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:00.241673 32048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be7b48776054b60bebbaa7347c3c5a6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37885 } }
I20260806 04:56:00.242151 32048 leader_election.cc:302] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0be7b48776054b60bebbaa7347c3c5a6; no voters:
I20260806 04:56:00.242751 32048 leader_election.cc:288] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:00.242931 32053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:00.243633 32053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [term 1 LEADER]: Becoming Leader. State: Replica: 0be7b48776054b60bebbaa7347c3c5a6, State: Running, Role: LEADER
I20260806 04:56:00.244163 32053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be7b48776054b60bebbaa7347c3c5a6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37885 } }
I20260806 04:56:00.244918 32048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:00.246999 32055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0be7b48776054b60bebbaa7347c3c5a6. Latest consensus state: current_term: 1 leader_uuid: "0be7b48776054b60bebbaa7347c3c5a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be7b48776054b60bebbaa7347c3c5a6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37885 } } }
I20260806 04:56:00.247473 32055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:00.247099 32054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0be7b48776054b60bebbaa7347c3c5a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be7b48776054b60bebbaa7347c3c5a6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37885 } } }
I20260806 04:56:00.248377 32054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:00.249220 32063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:00.253772 32063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:00.262389 32063 catalog_manager.cc:1347] Generated new cluster ID: 1fdf5eff4e7f494fa73b20a650c5145c
I20260806 04:56:00.262490 32063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:00.285095 32063 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:00.286311 32063 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:00.298132 32063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6: Generated new TSK 0
I20260806 04:56:00.299109 32063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:00.313808 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:37885
--builtin_ntp_servers=127.13.148.148:43729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:56:00.632186 32073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:00.632505 32073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:00.632664 32073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:00.639631 32073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:00.639786 32073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:00.639933 32073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:56:00.648078 32073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:43729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:37885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:00.649422 32073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:00.651189 32073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:00.663734 32081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:00.663787 32079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:00.663980 32078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:00.664664 32073 server_base.cc:1034] running on GCE node
I20260806 04:56:00.665275 32073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:00.666109 32073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:00.667294 32073 hybrid_clock.cc:648] HybridClock initialized: now 1785992160667242 us; error 42 us; skew 500 ppm
I20260806 04:56:00.667615 32073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:56:00.669845 32073 webserver.cc:460] Webserver started at http://127.13.148.129:41905/ using document root <none> and password file <none>
I20260806 04:56:00.670534 32073 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:00.670639 32073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:00.670930 32073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:00.673604 32073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "3b51a97a69cd49449db2ff5fd394554f"
format_stamp: "Formatted at 2026-08-06 04:56:00 on dist-test-slave-1x4g"
server_key: "144a45fc6327374516699a90431e0fae"
server_key_iv: "0e4893eb686f087883054b281a6b6314"
server_key_version: "encryptionkey@0"
I20260806 04:56:00.674292 32073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "3b51a97a69cd49449db2ff5fd394554f"
format_stamp: "Formatted at 2026-08-06 04:56:00 on dist-test-slave-1x4g"
server_key: "144a45fc6327374516699a90431e0fae"
server_key_iv: "0e4893eb686f087883054b281a6b6314"
server_key_version: "encryptionkey@0"
I20260806 04:56:00.679138 32073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 04:56:00.682246 32087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:00.683413 32073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:56:00.683576 32073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "3b51a97a69cd49449db2ff5fd394554f"
format_stamp: "Formatted at 2026-08-06 04:56:00 on dist-test-slave-1x4g"
server_key: "144a45fc6327374516699a90431e0fae"
server_key_iv: "0e4893eb686f087883054b281a6b6314"
server_key_version: "encryptionkey@0"
I20260806 04:56:00.683763 32073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:00.710187 32073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:00.711009 32073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:00.711300 32073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:00.712246 32073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:56:00.713573 32073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:56:00.713676 32073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:00.713794 32073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:56:00.713881 32073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:00.745571 32073 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:36731
I20260806 04:56:00.745613 32201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:36731 every 8 connection(s)
I20260806 04:56:00.746829 32073 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:56:00.752038 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 32073
I20260806 04:56:00.752207 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:56:00.752703 13906 external_mini_cluster.cc:1428] Setting key 3e606fd6490d1d6f3c43b0ba69342584
I20260806 04:56:00.755445 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:37885
--builtin_ntp_servers=127.13.148.148:43729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:56:00.760303 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:00.749511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:43535 (local address 127.13.148.190:37885)
0806 04:56:00.749909 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:00.749913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:00.750901 (+ 988us) server_negotiation.cc:407] Connection header received
0806 04:56:00.753014 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:00.753017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:00.753101 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:00.753200 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:00.754570 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:00.755157 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:00.757532 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:00.757949 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:00.758053 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:00.759611 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:00.759616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:00.759618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:00.759634 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:00.759674 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:00.759685 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:00.759751 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:00.759981 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 04:56:00.760058 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":218,"threads_started":1}
I20260806 04:56:00.761754 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:00.749945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:37885 (local address 127.13.148.129:43535)
0806 04:56:00.750751 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:00.750794 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:00.752720 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:00.753423 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:00.753430 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:00.753506 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:00.754194 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:00.754215 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:00.755300 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:00.755305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:00.757384 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:00.757404 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:00.758182 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:00.758191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:00.758518 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:00.759287 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:56:00.759292 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:00.759342 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:56:00.759436 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:00.759772 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:00.759784 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:00.759793 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 04:56:00.759967 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 04:56:00.760380 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":233,"threads_started":1}
I20260806 04:56:00.763240 32202 heartbeater.cc:347] Connected to a master server at 127.13.148.190:37885
I20260806 04:56:00.763554 32202 heartbeater.cc:464] Registering TS with master...
I20260806 04:56:00.764452 32202 heartbeater.cc:511] Master 127.13.148.190:37885 requested a full tablet report, sending...
I20260806 04:56:00.766826 32011 ts_manager.cc:194] Registered new tserver with Master: 3b51a97a69cd49449db2ff5fd394554f (127.13.148.129:36731)
I20260806 04:56:00.769140 32011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.129:43535
W20260806 04:56:01.080878 32206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:01.081148 32206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:01.081280 32206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:01.087982 32206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:01.088101 32206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:01.088269 32206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:56:01.096104 32206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:43729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:37885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:01.097456 32206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:01.099421 32206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:01.111572 32214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:01.111634 32212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:01.111651 32211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:01.112255 32206 server_base.cc:1034] running on GCE node
I20260806 04:56:01.112717 32206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:01.113518 32206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:01.114698 32206 hybrid_clock.cc:648] HybridClock initialized: now 1785992161114664 us; error 29 us; skew 500 ppm
I20260806 04:56:01.114987 32206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:56:01.117007 32206 webserver.cc:460] Webserver started at http://127.13.148.130:40091/ using document root <none> and password file <none>
I20260806 04:56:01.117712 32206 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:01.117830 32206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:01.118151 32206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:01.120924 32206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "2213b910ad8d43d1a1669fd5e0b086b0"
format_stamp: "Formatted at 2026-08-06 04:56:01 on dist-test-slave-1x4g"
server_key: "0fe704d6ad5c9cbd0c5b15b29d0032be"
server_key_iv: "09982ee1d77338201aaed54d88f94cb5"
server_key_version: "encryptionkey@0"
I20260806 04:56:01.121721 32206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "2213b910ad8d43d1a1669fd5e0b086b0"
format_stamp: "Formatted at 2026-08-06 04:56:01 on dist-test-slave-1x4g"
server_key: "0fe704d6ad5c9cbd0c5b15b29d0032be"
server_key_iv: "09982ee1d77338201aaed54d88f94cb5"
server_key_version: "encryptionkey@0"
I20260806 04:56:01.126531 32206 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 04:56:01.129649 32220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:01.130743 32206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:56:01.130903 32206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "2213b910ad8d43d1a1669fd5e0b086b0"
format_stamp: "Formatted at 2026-08-06 04:56:01 on dist-test-slave-1x4g"
server_key: "0fe704d6ad5c9cbd0c5b15b29d0032be"
server_key_iv: "09982ee1d77338201aaed54d88f94cb5"
server_key_version: "encryptionkey@0"
I20260806 04:56:01.131099 32206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:01.144752 32206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:01.145517 32206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:01.145798 32206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:01.146692 32206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:56:01.148002 32206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:56:01.148115 32206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:01.148248 32206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:56:01.148356 32206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:01.177889 32206 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:46043
I20260806 04:56:01.177942 32336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:46043 every 8 connection(s)
I20260806 04:56:01.179194 32206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:56:01.185247 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 32206
I20260806 04:56:01.185459 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:56:01.185916 13906 external_mini_cluster.cc:1428] Setting key 25cd2efc8776b69726713f98b72a1894
I20260806 04:56:01.188694 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:37885
--builtin_ntp_servers=127.13.148.148:43729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:56:01.191279 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:01.181615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:60455 (local address 127.13.148.190:37885)
0806 04:56:01.181838 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:01.181844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:01.183172 (+ 1328us) server_negotiation.cc:407] Connection header received
0806 04:56:01.184643 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:01.184647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:01.184707 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:01.184825 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:01.185956 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.186870 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.188569 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.189229 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.189363 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:01.190497 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:01.190505 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:01.190511 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:01.190532 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:01.190571 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:01.190587 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:01.190668 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:01.190842 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 04:56:01.190939 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 04:56:01.192063 32339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:01.182175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:37885 (local address 127.13.148.130:60455)
0806 04:56:01.183019 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:01.183052 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:01.184422 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:01.185025 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:01.185032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:01.185105 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:01.185776 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:01.185796 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.187031 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.187035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:01.188412 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:01.188425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.189408 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.189412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:01.189584 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:01.190219 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:56:01.190221 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:01.190255 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:56:01.190319 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:01.190671 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:01.190678 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:01.190682 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 04:56:01.190799 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 04:56:01.191048 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":254,"threads_started":1}
I20260806 04:56:01.193454 32337 heartbeater.cc:347] Connected to a master server at 127.13.148.190:37885
I20260806 04:56:01.193737 32337 heartbeater.cc:464] Registering TS with master...
I20260806 04:56:01.194417 32337 heartbeater.cc:511] Master 127.13.148.190:37885 requested a full tablet report, sending...
I20260806 04:56:01.195686 32011 ts_manager.cc:194] Registered new tserver with Master: 2213b910ad8d43d1a1669fd5e0b086b0 (127.13.148.130:46043)
I20260806 04:56:01.196425 32011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.130:60455
W20260806 04:56:01.500458 32340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:01.500705 32340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:01.500823 32340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:01.507550 32340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:01.507656 32340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:01.507778 32340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:56:01.516144 32340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:43729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:37885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:01.517445 32340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:01.519132 32340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:01.531435 32345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:01.531437 32348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:01.531462 32346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:01.531896 32340 server_base.cc:1034] running on GCE node
I20260806 04:56:01.532590 32340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:01.533483 32340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:01.534672 32340 hybrid_clock.cc:648] HybridClock initialized: now 1785992161534641 us; error 39 us; skew 500 ppm
I20260806 04:56:01.534986 32340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:56:01.537175 32340 webserver.cc:460] Webserver started at http://127.13.148.131:45835/ using document root <none> and password file <none>
I20260806 04:56:01.537858 32340 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:01.537998 32340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:01.538313 32340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:01.541043 32340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "2bfeaab611914e55a201ede49838faea"
format_stamp: "Formatted at 2026-08-06 04:56:01 on dist-test-slave-1x4g"
server_key: "c0c249f396375e2f3255095069b32d50"
server_key_iv: "595776692fcf2b5518bd0b4c8e7cacc2"
server_key_version: "encryptionkey@0"
I20260806 04:56:01.541733 32340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "2bfeaab611914e55a201ede49838faea"
format_stamp: "Formatted at 2026-08-06 04:56:01 on dist-test-slave-1x4g"
server_key: "c0c249f396375e2f3255095069b32d50"
server_key_iv: "595776692fcf2b5518bd0b4c8e7cacc2"
server_key_version: "encryptionkey@0"
I20260806 04:56:01.546445 32340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:56:01.549810 32354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:01.550940 32340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260806 04:56:01.551103 32340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "2bfeaab611914e55a201ede49838faea"
format_stamp: "Formatted at 2026-08-06 04:56:01 on dist-test-slave-1x4g"
server_key: "c0c249f396375e2f3255095069b32d50"
server_key_iv: "595776692fcf2b5518bd0b4c8e7cacc2"
server_key_version: "encryptionkey@0"
I20260806 04:56:01.551293 32340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:01.583178 32340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:01.583973 32340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:01.584261 32340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:01.585258 32340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:56:01.586640 32340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:56:01.586755 32340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:01.586879 32340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:56:01.586961 32340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:01.618155 32340 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:42659
I20260806 04:56:01.618254 32469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:42659 every 8 connection(s)
I20260806 04:56:01.619542 32340 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:56:01.625768 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 32340
I20260806 04:56:01.625959 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:56:01.626415 13906 external_mini_cluster.cc:1428] Setting key eae863d9bc1d7405187f237a4399077a
I20260806 04:56:01.632371 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:01.622124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:57645 (local address 127.13.148.190:37885)
0806 04:56:01.622324 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:01.622329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:01.623698 (+ 1369us) server_negotiation.cc:407] Connection header received
0806 04:56:01.625534 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:01.625541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:01.625636 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:01.625753 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:01.626917 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.627984 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.629995 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.630388 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.630481 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:01.631765 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:01.631769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:01.631771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:01.631787 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:01.631813 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:01.631823 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:01.631884 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:01.632093 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 04:56:01.632164 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 04:56:01.633386 32472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:01.622693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:37885 (local address 127.13.148.131:57645)
0806 04:56:01.623553 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:01.623589 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:01.625254 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:01.625977 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:01.625986 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:01.626112 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:01.626744 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:01.626762 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.628171 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.628177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:01.629862 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:01.629873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.630574 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.630596 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:01.630788 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:01.631482 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:56:01.631485 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:01.631524 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:56:01.631603 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:01.631899 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:01.631906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:01.631911 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 04:56:01.632040 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 04:56:01.632304 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":248,"threads_started":1}
I20260806 04:56:01.634696 32470 heartbeater.cc:347] Connected to a master server at 127.13.148.190:37885
I20260806 04:56:01.634941 32470 heartbeater.cc:464] Registering TS with master...
I20260806 04:56:01.635664 32470 heartbeater.cc:511] Master 127.13.148.190:37885 requested a full tablet report, sending...
I20260806 04:56:01.636919 32011 ts_manager.cc:194] Registered new tserver with Master: 2bfeaab611914e55a201ede49838faea (127.13.148.131:42659)
I20260806 04:56:01.637706 32011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.131:57645
I20260806 04:56:01.641425 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:56:01.650457 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:01.645187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.13.148.190:37885)
0806 04:56:01.645386 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:01.645394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:01.645726 (+ 332us) server_negotiation.cc:407] Connection header received
0806 04:56:01.645848 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:01.645851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:01.645952 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:01.646029 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:01.646414 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.647023 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.648421 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.648875 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.648961 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:01.649899 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:01.649903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:01.649905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:01.649920 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:01.649944 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:01.649954 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:01.650016 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:01.650114 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 04:56:01.650179 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:56:01.653771 32011 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:56:01.660429 32011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:56:01.684127 32482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:01.671474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36731 (local address 127.0.0.1:38376)
0806 04:56:01.672308 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:01.672344 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:01.672593 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:01.674072 (+ 1479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:01.674079 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:01.674112 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:01.674551 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:01.674565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.676886 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.676894 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:01.679903 (+ 3009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:01.679922 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.682864 (+ 2942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.682872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:01.683201 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:01.683268 (+ 67us) client_negotiation.cc:769] Sending connection context
0806 04:56:01.683434 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 04:56:01.683677 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":346,"threads_started":1}
I20260806 04:56:01.685145 32483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:01.671555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38376 (local address 127.13.148.129:36731)
0806 04:56:01.673052 (+ 1497us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:01.673064 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:01.673102 (+ 38us) server_negotiation.cc:407] Connection header received
0806 04:56:01.673222 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:01.673234 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:01.673519 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:01.673728 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:01.674786 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.676666 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.680206 (+ 3540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.682257 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.682435 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:01.682711 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:01.684570 (+ 1859us) server_negotiation.cc:299] Negotiation successful
0806 04:56:01.684809 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1269,"thread_start_us":1279,"threads_started":1}
I20260806 04:56:01.686283 32484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:01.671648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:46043 (local address 127.0.0.1:42872)
0806 04:56:01.674308 (+ 2660us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:01.674328 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:01.674465 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:01.675506 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:01.675526 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:01.675559 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:01.675983 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:01.675996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.679753 (+ 3757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.679759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:01.682447 (+ 2688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:01.682466 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.685443 (+ 2977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.685452 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:01.685712 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:01.685762 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 04:56:01.685858 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 04:56:01.685987 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2506,"thread_start_us":2821,"threads_started":1}
I20260806 04:56:01.686942 32485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:01.672266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42872 (local address 127.13.148.130:46043)
0806 04:56:01.673859 (+ 1593us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:01.673869 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:01.674625 (+ 756us) server_negotiation.cc:407] Connection header received
0806 04:56:01.674746 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:01.674755 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:01.674980 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:01.675162 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:01.676178 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.679554 (+ 3376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.682722 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.684693 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.684925 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:01.685175 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:01.686501 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0806 04:56:01.686700 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1446,"thread_start_us":1307,"threads_started":1}
I20260806 04:56:01.689234 32136 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5fae1bf0e641dd90e654b1de64e41d (DEFAULT_TABLE table=test-table [id=f9b9c73f4fc3480497499329e1bac1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:56:01.690653 32136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b5fae1bf0e641dd90e654b1de64e41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:01.690759 32270 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5fae1bf0e641dd90e654b1de64e41d (DEFAULT_TABLE table=test-table [id=f9b9c73f4fc3480497499329e1bac1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:56:01.691797 32270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b5fae1bf0e641dd90e654b1de64e41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:01.694700 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:01.673628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:42659 (local address 127.0.0.1:55176)
0806 04:56:01.685944 (+ 12316us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:01.685967 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:01.686090 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:01.688049 (+ 1959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:01.688052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:01.688072 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:01.688532 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:01.688545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.690317 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.690322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:01.691968 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:01.691983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.694000 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.694007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:01.694177 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:01.694209 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 04:56:01.694305 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 04:56:01.694429 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12191,"thread_start_us":12030,"threads_started":1}
I20260806 04:56:01.694859 32487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:01.674202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.13.148.131:42659)
0806 04:56:01.685205 (+ 11003us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:01.685212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:01.687382 (+ 2170us) server_negotiation.cc:407] Connection header received
0806 04:56:01.687504 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:01.687515 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:01.687704 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:01.687832 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:01.688751 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.690159 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.692155 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.693496 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.693691 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:01.693880 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:01.694366 (+ 486us) server_negotiation.cc:299] Negotiation successful
0806 04:56:01.694534 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10832,"thread_start_us":11127,"threads_started":1}
I20260806 04:56:01.698616 32403 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5fae1bf0e641dd90e654b1de64e41d (DEFAULT_TABLE table=test-table [id=f9b9c73f4fc3480497499329e1bac1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:56:01.699928 32403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b5fae1bf0e641dd90e654b1de64e41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:01.702744 32488 tablet_bootstrap.cc:492] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0: Bootstrap starting.
I20260806 04:56:01.705927 32488 tablet_bootstrap.cc:654] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:01.707075 32488 log.cc:824] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:01.709666 32489 tablet_bootstrap.cc:492] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f: Bootstrap starting.
I20260806 04:56:01.712968 32488 tablet_bootstrap.cc:492] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0: No bootstrap required, opened a new log
I20260806 04:56:01.713233 32488 ts_tablet_manager.cc:1397] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260806 04:56:01.713263 32489 tablet_bootstrap.cc:654] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:01.714228 32489 log.cc:824] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:01.715296 32491 tablet_bootstrap.cc:492] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea: Bootstrap starting.
I20260806 04:56:01.718964 32491 tablet_bootstrap.cc:654] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:01.719482 32489 tablet_bootstrap.cc:492] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f: No bootstrap required, opened a new log
I20260806 04:56:01.719731 32489 ts_tablet_manager.cc:1397] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260806 04:56:01.720378 32491 log.cc:824] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:01.721843 32488 raft_consensus.cc:348] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b51a97a69cd49449db2ff5fd394554f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36731 } } peers { permanent_uuid: "2213b910ad8d43d1a1669fd5e0b086b0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46043 } } peers { permanent_uuid: "2bfeaab611914e55a201ede49838faea" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 42659 } }
I20260806 04:56:01.722187 32488 raft_consensus.cc:374] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:01.722318 32488 raft_consensus.cc:729] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2213b910ad8d43d1a1669fd5e0b086b0, State: Initialized, Role: FOLLOWER
I20260806 04:56:01.722887 32488 consensus_queue.cc:260] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b51a97a69cd49449db2ff5fd394554f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36731 } } peers { permanent_uuid: "2213b910ad8d43d1a1669fd5e0b086b0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46043 } } peers { permanent_uuid: "2bfeaab611914e55a201ede49838faea" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 42659 } }
I20260806 04:56:01.723266 32491 tablet_bootstrap.cc:492] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea: No bootstrap required, opened a new log
I20260806 04:56:01.723524 32491 ts_tablet_manager.cc:1397] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 04:56:01.724022 32337 heartbeater.cc:503] Master 127.13.148.190:37885 was elected leader, sending a full tablet report...
I20260806 04:56:01.724465 32488 ts_tablet_manager.cc:1428] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260806 04:56:01.729319 32489 raft_consensus.cc:348] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b51a97a69cd49449db2ff5fd394554f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36731 } } peers { permanent_uuid: "2213b910ad8d43d1a1669fd5e0b086b0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46043 } } peers { permanent_uuid: "2bfeaab611914e55a201ede49838faea" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 42659 } }
I20260806 04:56:01.729732 32489 raft_consensus.cc:374] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:01.729825 32489 raft_consensus.cc:729] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b51a97a69cd49449db2ff5fd394554f, State: Initialized, Role: FOLLOWER
I20260806 04:56:01.730417 32489 consensus_queue.cc:260] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b51a97a69cd49449db2ff5fd394554f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36731 } } peers { permanent_uuid: "2213b910ad8d43d1a1669fd5e0b086b0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46043 } } peers { permanent_uuid: "2bfeaab611914e55a201ede49838faea" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 42659 } }
I20260806 04:56:01.731468 32202 heartbeater.cc:503] Master 127.13.148.190:37885 was elected leader, sending a full tablet report...
I20260806 04:56:01.731910 32489 ts_tablet_manager.cc:1428] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260806 04:56:01.733752 32491 raft_consensus.cc:348] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b51a97a69cd49449db2ff5fd394554f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36731 } } peers { permanent_uuid: "2213b910ad8d43d1a1669fd5e0b086b0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46043 } } peers { permanent_uuid: "2bfeaab611914e55a201ede49838faea" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 42659 } }
I20260806 04:56:01.734064 32491 raft_consensus.cc:374] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:01.734155 32491 raft_consensus.cc:729] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bfeaab611914e55a201ede49838faea, State: Initialized, Role: FOLLOWER
I20260806 04:56:01.734776 32491 consensus_queue.cc:260] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b51a97a69cd49449db2ff5fd394554f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36731 } } peers { permanent_uuid: "2213b910ad8d43d1a1669fd5e0b086b0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46043 } } peers { permanent_uuid: "2bfeaab611914e55a201ede49838faea" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 42659 } }
I20260806 04:56:01.735742 32470 heartbeater.cc:503] Master 127.13.148.190:37885 was elected leader, sending a full tablet report...
I20260806 04:56:01.736173 32491 ts_tablet_manager.cc:1428] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260806 04:56:01.749120 32203 tablet.cc:2369] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:56:01.773350 32494 raft_consensus.cc:482] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:56:01.773626 32494 raft_consensus.cc:504] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b51a97a69cd49449db2ff5fd394554f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36731 } } peers { permanent_uuid: "2213b910ad8d43d1a1669fd5e0b086b0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46043 } } peers { permanent_uuid: "2bfeaab611914e55a201ede49838faea" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 42659 } }
I20260806 04:56:01.775045 32494 leader_election.cc:288] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b51a97a69cd49449db2ff5fd394554f (127.13.148.129:36731), 2bfeaab611914e55a201ede49838faea (127.13.148.131:42659)
I20260806 04:56:01.781541 32483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:01.775547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:49073 (local address 127.13.148.129:36731)
0806 04:56:01.775756 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:01.775763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:01.776188 (+ 425us) server_negotiation.cc:407] Connection header received
0806 04:56:01.776404 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:01.776412 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:01.776474 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:01.776577 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:01.777220 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.777999 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.779934 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.780688 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.780872 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:01.780936 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:01.781205 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 04:56:01.781317 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 04:56:01.781635 32497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:01.775297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36731 (local address 127.13.148.130:49073)
0806 04:56:01.776079 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:01.776093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:01.776271 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:01.776611 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:01.776615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:01.776658 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:01.777075 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:01.777089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.778150 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.778154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:01.779752 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:01.779764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.780888 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.780893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:01.781046 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:01.781068 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:56:01.781168 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 04:56:01.781289 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":1,"thread_start_us":432,"threads_started":1}
I20260806 04:56:01.781929 32487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:01.775530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:40177 (local address 127.13.148.131:42659)
0806 04:56:01.775927 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:01.775934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:01.776483 (+ 549us) server_negotiation.cc:407] Connection header received
0806 04:56:01.776718 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:01.776727 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:01.776839 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:01.776933 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:01.777482 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.778472 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.780299 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.781103 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.781285 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:01.781348 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:01.781602 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 04:56:01.781714 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260806 04:56:01.782038 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:01.775401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:42659 (local address 127.13.148.130:40177)
0806 04:56:01.776374 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:01.776387 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:01.776539 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:01.776956 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:01.776960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:01.776986 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:01.777329 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:01.777337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.778741 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.778745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:01.780115 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:01.780127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.781286 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.781291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:01.781453 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:01.781475 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:56:01.781578 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 04:56:01.781698 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":30,"thread_start_us":694,"threads_started":1}
I20260806 04:56:01.782817 32156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5fae1bf0e641dd90e654b1de64e41d" candidate_uuid: "2213b910ad8d43d1a1669fd5e0b086b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b51a97a69cd49449db2ff5fd394554f" is_pre_election: true
I20260806 04:56:01.782932 32423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5fae1bf0e641dd90e654b1de64e41d" candidate_uuid: "2213b910ad8d43d1a1669fd5e0b086b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bfeaab611914e55a201ede49838faea" is_pre_election: true
I20260806 04:56:01.783319 32156 raft_consensus.cc:2454] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2213b910ad8d43d1a1669fd5e0b086b0 in term 0.
I20260806 04:56:01.783396 32423 raft_consensus.cc:2454] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2213b910ad8d43d1a1669fd5e0b086b0 in term 0.
I20260806 04:56:01.783985 32223 leader_election.cc:302] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2213b910ad8d43d1a1669fd5e0b086b0, 3b51a97a69cd49449db2ff5fd394554f; no voters:
I20260806 04:56:01.784359 32494 raft_consensus.cc:2781] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:56:01.784489 32494 raft_consensus.cc:482] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:56:01.784601 32494 raft_consensus.cc:3037] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:01.786756 32494 raft_consensus.cc:504] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b51a97a69cd49449db2ff5fd394554f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36731 } } peers { permanent_uuid: "2213b910ad8d43d1a1669fd5e0b086b0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46043 } } peers { permanent_uuid: "2bfeaab611914e55a201ede49838faea" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 42659 } }
I20260806 04:56:01.787380 32494 leader_election.cc:288] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [CANDIDATE]: Term 1 election: Requested vote from peers 3b51a97a69cd49449db2ff5fd394554f (127.13.148.129:36731), 2bfeaab611914e55a201ede49838faea (127.13.148.131:42659)
I20260806 04:56:01.787815 32156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5fae1bf0e641dd90e654b1de64e41d" candidate_uuid: "2213b910ad8d43d1a1669fd5e0b086b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b51a97a69cd49449db2ff5fd394554f"
I20260806 04:56:01.787863 32423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5fae1bf0e641dd90e654b1de64e41d" candidate_uuid: "2213b910ad8d43d1a1669fd5e0b086b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bfeaab611914e55a201ede49838faea"
I20260806 04:56:01.788043 32423 raft_consensus.cc:3037] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:01.788030 32156 raft_consensus.cc:3037] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:01.791097 32156 raft_consensus.cc:2454] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2213b910ad8d43d1a1669fd5e0b086b0 in term 1.
I20260806 04:56:01.791105 32423 raft_consensus.cc:2454] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2213b910ad8d43d1a1669fd5e0b086b0 in term 1.
I20260806 04:56:01.791471 32223 leader_election.cc:302] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2213b910ad8d43d1a1669fd5e0b086b0, 3b51a97a69cd49449db2ff5fd394554f; no voters:
I20260806 04:56:01.791778 32494 raft_consensus.cc:2781] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:01.792251 32494 raft_consensus.cc:686] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [term 1 LEADER]: Becoming Leader. State: Replica: 2213b910ad8d43d1a1669fd5e0b086b0, State: Running, Role: LEADER
I20260806 04:56:01.792683 32494 consensus_queue.cc:237] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b51a97a69cd49449db2ff5fd394554f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36731 } } peers { permanent_uuid: "2213b910ad8d43d1a1669fd5e0b086b0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46043 } } peers { permanent_uuid: "2bfeaab611914e55a201ede49838faea" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 42659 } }
I20260806 04:56:01.797363 32011 catalog_manager.cc:5495] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2213b910ad8d43d1a1669fd5e0b086b0 (127.13.148.130). New cstate: current_term: 1 leader_uuid: "2213b910ad8d43d1a1669fd5e0b086b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b51a97a69cd49449db2ff5fd394554f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2213b910ad8d43d1a1669fd5e0b086b0" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 46043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bfeaab611914e55a201ede49838faea" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 42659 } health_report { overall_health: UNKNOWN } } }
I20260806 04:56:01.812832 32485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:01.808019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.13.148.130:46043)
0806 04:56:01.808194 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:01.808199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:01.808215 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:56:01.808360 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:01.808364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:01.808434 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:01.808557 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:01.808937 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.809617 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.810907 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.811289 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.811376 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:01.811796 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:01.812254 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:01.812356 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:01.812441 (+ 85us) server_negotiation.cc:299] Negotiation successful
0806 04:56:01.812500 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 04:56:01.820427 32423 raft_consensus.cc:1261] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea [term 1 FOLLOWER]: Refusing update from remote peer 2213b910ad8d43d1a1669fd5e0b086b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:56:01.820463 32156 raft_consensus.cc:1261] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f [term 1 FOLLOWER]: Refusing update from remote peer 2213b910ad8d43d1a1669fd5e0b086b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:56:01.821414 32499 consensus_queue.cc:1035] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b51a97a69cd49449db2ff5fd394554f" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:56:01.821730 32494 consensus_queue.cc:1035] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bfeaab611914e55a201ede49838faea" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 42659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:56:01.853289 32010 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:56:01.853614 32010 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:56:01.857554 32010 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0be7b48776054b60bebbaa7347c3c5a6: Sending DeleteTablet for 3 replicas of tablet 2b5fae1bf0e641dd90e654b1de64e41d
I20260806 04:56:01.858726 32270 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b5fae1bf0e641dd90e654b1de64e41d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:56:01 UTC) from {username='slave'} at 127.0.0.1:42872
I20260806 04:56:01.858718 32136 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b5fae1bf0e641dd90e654b1de64e41d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:56:01 UTC) from {username='slave'} at 127.0.0.1:38376
I20260806 04:56:01.859572 32403 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b5fae1bf0e641dd90e654b1de64e41d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:56:01 UTC) from {username='slave'} at 127.0.0.1:55176
I20260806 04:56:01.860602 32511 tablet_replica.cc:330] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f: stopping tablet replica
I20260806 04:56:01.860694 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 32073
I20260806 04:56:01.861208 32512 tablet_replica.cc:330] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0: stopping tablet replica
I20260806 04:56:01.861739 32512 raft_consensus.cc:2229] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:56:01.862310 32511 raft_consensus.cc:2229] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:56:01.863168 32512 raft_consensus.cc:2258] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:56:01.864012 32511 raft_consensus.cc:2258] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:56:01.865939 32512 ts_tablet_manager.cc:1905] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:56:01.867908 32511 ts_tablet_manager.cc:1905] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:56:01.874279 32512 ts_tablet_manager.cc:1918] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:56:01.874519 32512 log.cc:1196] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/2b5fae1bf0e641dd90e654b1de64e41d
I20260806 04:56:01.875134 32512 ts_tablet_manager.cc:1939] T 2b5fae1bf0e641dd90e654b1de64e41d P 2213b910ad8d43d1a1669fd5e0b086b0: Deleting consensus metadata
I20260806 04:56:01.877118 31993 catalog_manager.cc:4841] TS 2213b910ad8d43d1a1669fd5e0b086b0 (127.13.148.130:46043): tablet 2b5fae1bf0e641dd90e654b1de64e41d (table test-table [id=f9b9c73f4fc3480497499329e1bac1c9]) successfully deleted
I20260806 04:56:01.878299 32483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:01.861928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38386 (local address 127.13.148.129:36731)
0806 04:56:01.862104 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:01.862110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:01.869804 (+ 7694us) server_negotiation.cc:407] Connection header received
0806 04:56:01.870061 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:01.870067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:01.870178 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:01.870268 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:01.870965 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.872136 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.874206 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:01.874833 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:01.874929 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:01.876434 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:01.876446 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:01.876467 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:01.877324 (+ 857us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:01.877389 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:01.877414 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:01.877522 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:01.877794 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 04:56:01.877906 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:56:01.879185 32197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:01.879230 32511 ts_tablet_manager.cc:1918] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:56:01.879375 32511 log.cc:1196] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/2b5fae1bf0e641dd90e654b1de64e41d
I20260806 04:56:01.881572 32513 tablet_replica.cc:330] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea: stopping tablet replica
I20260806 04:56:01.882176 32513 raft_consensus.cc:2229] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:56:01.882836 32513 raft_consensus.cc:2258] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:56:01.885103 32513 ts_tablet_manager.cc:1905] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:56:01.889632 32513 ts_tablet_manager.cc:1918] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:56:01.889799 32513 log.cc:1196] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/2b5fae1bf0e641dd90e654b1de64e41d
I20260806 04:56:01.890237 32513 ts_tablet_manager.cc:1939] T 2b5fae1bf0e641dd90e654b1de64e41d P 2bfeaab611914e55a201ede49838faea: Deleting consensus metadata
I20260806 04:56:01.891708 31996 catalog_manager.cc:4841] TS 2bfeaab611914e55a201ede49838faea (127.13.148.131:42659): tablet 2b5fae1bf0e641dd90e654b1de64e41d (table test-table [id=f9b9c73f4fc3480497499329e1bac1c9]) successfully deleted
I20260806 04:56:01.980448 32511 ts_tablet_manager.cc:1939] T 2b5fae1bf0e641dd90e654b1de64e41d P 3b51a97a69cd49449db2ff5fd394554f: Deleting consensus metadata
I20260806 04:56:01.980576 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 32073
W20260806 04:56:02.006227 31995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.129:36731 (error 108)
I20260806 04:56:02.006970 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 32206
I20260806 04:56:02.008554 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:02.007824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36731 (local address 127.0.0.1:38396)
0806 04:56:02.008257 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:02.008398 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:36731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304}
W20260806 04:56:02.008847 31995 catalog_manager.cc:4584] TS 3b51a97a69cd49449db2ff5fd394554f (127.13.148.129:36731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b5fae1bf0e641dd90e654b1de64e41d: Network error: Client connection negotiation failed: client connection to 127.13.148.129:36731: connect: Connection refused (error 111)
I20260806 04:56:02.014740 32485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:02.007872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42886 (local address 127.13.148.130:46043)
0806 04:56:02.008088 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:02.008094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:02.008243 (+ 149us) server_negotiation.cc:407] Connection header received
0806 04:56:02.008429 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:02.008435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:02.008571 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:02.008680 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:02.009318 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:02.010076 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:02.011794 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:02.012156 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:02.012238 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:02.013382 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:02.013390 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:02.013407 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:02.013992 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:02.014042 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:02.014060 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:02.014137 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:02.014299 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 04:56:02.014372 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 04:56:02.015470 32332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:02.062361 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:02.061908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36731 (local address 127.0.0.1:38404)
0806 04:56:02.062096 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:02.062178 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:36731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 04:56:02.063303 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:02.062994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36731 (local address 127.0.0.1:38414)
0806 04:56:02.063130 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:02.063197 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:36731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 04:56:02.117745 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 32206
I20260806 04:56:02.123986 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:02.123518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36731 (local address 127.0.0.1:38416)
0806 04:56:02.123738 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:02.123823 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:36731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 04:56:02.124939 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:02.124644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36731 (local address 127.0.0.1:38420)
0806 04:56:02.124763 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:02.124845 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:36731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 04:56:02.138381 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 32340
I20260806 04:56:02.144194 32487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:02.139031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.13.148.131:42659)
0806 04:56:02.139168 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:02.139172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:02.139187 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:56:02.139336 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:02.139340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:02.139418 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:02.139492 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:02.139817 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:02.140466 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:02.141686 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:02.142029 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:02.142113 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:02.143006 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:02.143021 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:02.143035 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:02.143537 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:02.143578 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:02.143601 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:02.143678 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:02.143827 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 04:56:02.143932 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:56:02.144768 32465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:02.205531 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:02.205091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36731 (local address 127.0.0.1:38422)
0806 04:56:02.205284 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:02.205378 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:36731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 04:56:02.206604 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:02.206247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36731 (local address 127.0.0.1:38436)
0806 04:56:02.206441 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:02.206513 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:36731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260806 04:56:02.230597 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 32340
I20260806 04:56:02.248299 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 31977
I20260806 04:56:02.249431 32043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:02.355316 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 31977
2026-08-06T04:56:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2624 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:02 dist-test-slave-1x4g krb5kdc[32522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:02 dist-test-slave-1x4g krb5kdc[32522](info): set up 2 sockets
Aug 06 04:56:02 dist-test-slave-1x4g krb5kdc[32522](info): commencing operation
krb5kdc: starting...
W20260806 04:56:03.406876 13906 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:03 dist-test-slave-1x4g krb5kdc[32522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:56:03.655861 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:44301
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:43093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:44301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:03.962649 32535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:03.962899 32535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:03.962981 32535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:03.969326 32535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:03.969420 32535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:03.969527 32535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:03.969617 32535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:03.969694 32535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:03.978082 32535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:43093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:44301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:44301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:03.979271 32535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:03.980458 32535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:03.991389 32542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:03.991365 32544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:03.991628 32541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:03.992229 32535 server_base.cc:1034] running on GCE node
I20260806 04:56:03.993028 32535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:03.994271 32535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:03.995489 32535 hybrid_clock.cc:648] HybridClock initialized: now 1785992163995455 us; error 47 us; skew 500 ppm
I20260806 04:56:03.995791 32535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:03 dist-test-slave-1x4g krb5kdc[32522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992163, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:03.998890 32535 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:56:04.000303 32535 webserver.cc:460] Webserver started at http://127.13.148.190:43567/ using document root <none> and password file <none>
I20260806 04:56:04.000986 32535 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:04.001118 32535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:04.001389 32535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:04.004237 32535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "6df5e4e8ba844a2ca935f4fb07cd435a"
format_stamp: "Formatted at 2026-08-06 04:56:03 on dist-test-slave-1x4g"
server_key: "c8c08904369695077f087d29555e2d26"
server_key_iv: "bdb3b2483668852c0f2d20604b1660f0"
server_key_version: "encryptionkey@0"
I20260806 04:56:04.005052 32535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "6df5e4e8ba844a2ca935f4fb07cd435a"
format_stamp: "Formatted at 2026-08-06 04:56:03 on dist-test-slave-1x4g"
server_key: "c8c08904369695077f087d29555e2d26"
server_key_iv: "bdb3b2483668852c0f2d20604b1660f0"
server_key_version: "encryptionkey@0"
I20260806 04:56:04.009804 32535 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:56:04.012931 32551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:04.013999 32535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:56:04.014168 32535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "6df5e4e8ba844a2ca935f4fb07cd435a"
format_stamp: "Formatted at 2026-08-06 04:56:03 on dist-test-slave-1x4g"
server_key: "c8c08904369695077f087d29555e2d26"
server_key_iv: "bdb3b2483668852c0f2d20604b1660f0"
server_key_version: "encryptionkey@0"
I20260806 04:56:04.014364 32535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:04.025833 32535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:04.029711 32535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:04.029970 32535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:04.046403 32535 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:44301
I20260806 04:56:04.046437 32603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:44301 every 8 connection(s)
I20260806 04:56:04.047668 32535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:04.048034 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 32535
I20260806 04:56:04.048213 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:04.048919 13906 external_mini_cluster.cc:1428] Setting key e2eaa32e1cbcbf2d552257037f74070c
I20260806 04:56:04.051937 32605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 04:56:04 dist-test-slave-1x4g krb5kdc[32522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:56:04.061362 32605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a: Bootstrap starting.
I20260806 04:56:04.064529 32605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:04.065623 32605 log.cc:824] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:04.068204 32605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a: No bootstrap required, opened a new log
I20260806 04:56:04.069180 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:04.050658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.13.148.190:44301)
0806 04:56:04.051367 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:04.051378 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:04.051413 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:56:04.052994 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:04.053020 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:04.053323 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:04.053649 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:04.054955 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:04.056155 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:04.057580 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:04.058220 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:04.058419 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:04.062255 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:04.062262 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:04.062274 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:04.062321 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:04.065315 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:04.066112 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:04.066118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:04.066123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:04.066287 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:04.066669 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:04.066673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:04.066680 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:04.067096 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:04.067375 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:04.067749 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 04:56:04.068166 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":343,"threads_started":1}
I20260806 04:56:04.075554 32605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df5e4e8ba844a2ca935f4fb07cd435a" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44301 } }
I20260806 04:56:04.075884 32605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:04.075989 32605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6df5e4e8ba844a2ca935f4fb07cd435a, State: Initialized, Role: FOLLOWER
I20260806 04:56:04.076458 32605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df5e4e8ba844a2ca935f4fb07cd435a" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44301 } }
I20260806 04:56:04.076745 32605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:04.076920 32605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:04.077112 32605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:04.079296 32605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df5e4e8ba844a2ca935f4fb07cd435a" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44301 } }
I20260806 04:56:04.079829 32605 leader_election.cc:302] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6df5e4e8ba844a2ca935f4fb07cd435a; no voters:
I20260806 04:56:04.080349 32605 leader_election.cc:288] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:04.080457 32610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:04.081164 32610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [term 1 LEADER]: Becoming Leader. State: Replica: 6df5e4e8ba844a2ca935f4fb07cd435a, State: Running, Role: LEADER
I20260806 04:56:04.081681 32610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df5e4e8ba844a2ca935f4fb07cd435a" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44301 } }
I20260806 04:56:04.082286 32605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:04.083910 32611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6df5e4e8ba844a2ca935f4fb07cd435a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df5e4e8ba844a2ca935f4fb07cd435a" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44301 } } }
I20260806 04:56:04.084122 32612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6df5e4e8ba844a2ca935f4fb07cd435a. Latest consensus state: current_term: 1 leader_uuid: "6df5e4e8ba844a2ca935f4fb07cd435a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df5e4e8ba844a2ca935f4fb07cd435a" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 44301 } } }
I20260806 04:56:04.084354 32612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:04.084295 32611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:04.089571 32615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:04.094316 32615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:04.104140 32615 catalog_manager.cc:1347] Generated new cluster ID: 78d95062f596456988a345bdca634d58
I20260806 04:56:04.104270 32615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:04.125010 32615 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:04.126266 32615 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:04.144001 32615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a: Generated new TSK 0
I20260806 04:56:04.145009 32615 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:56:04.211931 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44301
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:43093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:04.519343 32631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:04.519593 32631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:04.519738 32631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:04.526026 32631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:04.526111 32631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:04.526275 32631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:56:04.534886 32631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:43093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:04.535966 32631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:04.537247 32631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:04.549158 32636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:04.549134 32637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:04.549494 32639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:04.550441 32631 server_base.cc:1034] running on GCE node
I20260806 04:56:04.551051 32631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:04.551841 32631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:04.553076 32631 hybrid_clock.cc:648] HybridClock initialized: now 1785992164553023 us; error 42 us; skew 500 ppm
I20260806 04:56:04.553346 32631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:04 dist-test-slave-1x4g krb5kdc[32522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992164, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:04.559315 32631 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:56:04.560611 32631 webserver.cc:460] Webserver started at http://127.13.148.129:40921/ using document root <none> and password file <none>
I20260806 04:56:04.561250 32631 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:04.561388 32631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:04.561671 32631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:04.564116 32631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "f8428e53406e4646abc37c0ee38a69a3"
format_stamp: "Formatted at 2026-08-06 04:56:04 on dist-test-slave-1x4g"
server_key: "9d9463c8e8639788babada3d9df9af3e"
server_key_iv: "139e93cf5951ed4108f1c09dfc4cbc4a"
server_key_version: "encryptionkey@0"
I20260806 04:56:04.564831 32631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "f8428e53406e4646abc37c0ee38a69a3"
format_stamp: "Formatted at 2026-08-06 04:56:04 on dist-test-slave-1x4g"
server_key: "9d9463c8e8639788babada3d9df9af3e"
server_key_iv: "139e93cf5951ed4108f1c09dfc4cbc4a"
server_key_version: "encryptionkey@0"
I20260806 04:56:04.569492 32631 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 04:56:04.572638 32646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:04.573874 32631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:56:04.574038 32631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "f8428e53406e4646abc37c0ee38a69a3"
format_stamp: "Formatted at 2026-08-06 04:56:04 on dist-test-slave-1x4g"
server_key: "9d9463c8e8639788babada3d9df9af3e"
server_key_iv: "139e93cf5951ed4108f1c09dfc4cbc4a"
server_key_version: "encryptionkey@0"
I20260806 04:56:04.574218 32631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:04.584496 32631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:04.588323 32631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:04.588617 32631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:04.589552 32631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:56:04.590737 32631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:56:04.590826 32631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:04.590966 32631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:56:04.591030 32631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:04.621621 32631 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:46637
I20260806 04:56:04.621678 32761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:46637 every 8 connection(s)
I20260806 04:56:04.622828 32631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:56:04.623478 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 32631
I20260806 04:56:04.623670 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:56:04.624231 13906 external_mini_cluster.cc:1428] Setting key b7be49e2c249bda29090f017b7d38514
Aug 06 04:56:04 dist-test-slave-1x4g krb5kdc[32522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992164, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:56:04.641404 32764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:04.625421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:32843 (local address 127.13.148.190:44301)
0806 04:56:04.625874 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:04.625879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:04.626844 (+ 965us) server_negotiation.cc:407] Connection header received
0806 04:56:04.628600 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:04.628606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:04.628695 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:04.628850 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:04.630200 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:04.630833 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:04.632256 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:04.632660 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:04.632812 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:04.635905 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:04.635910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:04.635913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:04.635942 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:04.638565 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:04.639271 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:04.639278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:04.639286 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:04.639410 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:04.639833 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:04.639839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:04.639842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:04.640097 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:04.640244 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:04.640848 (+ 604us) server_negotiation.cc:299] Negotiation successful
0806 04:56:04.641062 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":228,"threads_started":1}
I20260806 04:56:04.642279 32765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:04.626028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:44301 (local address 127.13.148.129:32843)
0806 04:56:04.626708 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:04.626743 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:04.628328 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:04.629098 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:04.629105 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:04.629558 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:04.630042 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:04.630059 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:04.630995 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:04.630998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:04.632113 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:04.632124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:04.632893 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:04.632900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:04.633142 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:04.633778 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:04.633797 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:04.635693 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:04.638712 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:04.638721 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:04.638735 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:04.639142 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:04.639527 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:04.639532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:04.639536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:04.639728 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:04.640217 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:04.640223 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:04.640508 (+ 285us) client_negotiation.cc:769] Sending connection context
0806 04:56:04.640746 (+ 238us) client_negotiation.cc:240] Negotiation successful
0806 04:56:04.641212 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":316,"threads_started":1}
I20260806 04:56:04.643738 32762 heartbeater.cc:347] Connected to a master server at 127.13.148.190:44301
I20260806 04:56:04.644018 32762 heartbeater.cc:464] Registering TS with master...
I20260806 04:56:04.644709 32762 heartbeater.cc:511] Master 127.13.148.190:44301 requested a full tablet report, sending...
I20260806 04:56:04.646921 32568 ts_manager.cc:194] Registered new tserver with Master: f8428e53406e4646abc37c0ee38a69a3 (127.13.148.129:46637)
I20260806 04:56:04.648324 32568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:32843
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:56:04.681877 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44301
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:43093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:04.985915 300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:04.986181 300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:04.986269 300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:04.992606 300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:04.992691 300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:04.992799 300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:56:05.001024 300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:43093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:05.002045 300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:05.003124 300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:05.014593 305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:05.014675 306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:05.014693 308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:05.015585 300 server_base.cc:1034] running on GCE node
I20260806 04:56:05.016237 300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:05.017338 300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:05.018561 300 hybrid_clock.cc:648] HybridClock initialized: now 1785992165018520 us; error 50 us; skew 500 ppm
I20260806 04:56:05.018862 300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:05 dist-test-slave-1x4g krb5kdc[32522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992165, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:05.021878 300 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:56:05.023072 300 webserver.cc:460] Webserver started at http://127.13.148.130:39103/ using document root <none> and password file <none>
I20260806 04:56:05.023622 300 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:05.023710 300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:05.023929 300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:05.026500 300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "412cf425291a4bb0a4afa636dd40ccb3"
format_stamp: "Formatted at 2026-08-06 04:56:05 on dist-test-slave-1x4g"
server_key: "71cd4108d9045d36d2118335b068adb8"
server_key_iv: "8c1304756019afd8629cc38948bf8a48"
server_key_version: "encryptionkey@0"
I20260806 04:56:05.027120 300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "412cf425291a4bb0a4afa636dd40ccb3"
format_stamp: "Formatted at 2026-08-06 04:56:05 on dist-test-slave-1x4g"
server_key: "71cd4108d9045d36d2118335b068adb8"
server_key_iv: "8c1304756019afd8629cc38948bf8a48"
server_key_version: "encryptionkey@0"
I20260806 04:56:05.031591 300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:56:05.034621 315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:05.035778 300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:56:05.035944 300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "412cf425291a4bb0a4afa636dd40ccb3"
format_stamp: "Formatted at 2026-08-06 04:56:05 on dist-test-slave-1x4g"
server_key: "71cd4108d9045d36d2118335b068adb8"
server_key_iv: "8c1304756019afd8629cc38948bf8a48"
server_key_version: "encryptionkey@0"
I20260806 04:56:05.036108 300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:05.055580 300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:05.059779 300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:05.060058 300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:05.060956 300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:56:05.062112 300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:56:05.062228 300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:05.062361 300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:56:05.062426 300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:05.094578 300 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:40699
I20260806 04:56:05.094664 428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:40699 every 8 connection(s)
I20260806 04:56:05.095860 300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:56:05.105604 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 300
I20260806 04:56:05.105777 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:56:05.106309 13906 external_mini_cluster.cc:1428] Setting key 5be76b22f32e771cf83ba91f9a428792
Aug 06 04:56:05 dist-test-slave-1x4g krb5kdc[32522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992165, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:56:05.114962 32764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.098497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:47549 (local address 127.13.148.190:44301)
0806 04:56:05.098684 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:05.098688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:05.099808 (+ 1120us) server_negotiation.cc:407] Connection header received
0806 04:56:05.101758 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:05.101763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:05.101846 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:05.101948 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:05.103235 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.103882 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.105284 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.105654 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.105780 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.109583 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:05.109588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:05.109591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:05.109621 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:05.112010 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:05.112684 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:05.112694 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:05.112696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:05.112883 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:05.113297 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:05.113301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:05.113304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:05.113540 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:05.113696 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:05.114372 (+ 676us) server_negotiation.cc:299] Negotiation successful
0806 04:56:05.114607 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 04:56:05.115790 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.098991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:44301 (local address 127.13.148.130:47549)
0806 04:56:05.099650 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:05.099684 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:05.101458 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:05.102163 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:05.102170 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:05.102610 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:05.103078 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.103092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.104016 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.104021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.105148 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.105158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.105928 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.105935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.106207 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.106924 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:05.106965 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:05.109372 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:05.112143 (+ 2771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:05.112150 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:05.112163 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:05.112559 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:05.112987 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:05.112991 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:05.112995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:05.113180 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:05.113662 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:05.113670 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:05.114023 (+ 353us) client_negotiation.cc:769] Sending connection context
0806 04:56:05.114278 (+ 255us) client_negotiation.cc:240] Negotiation successful
0806 04:56:05.114758 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":248,"threads_started":1}
I20260806 04:56:05.117249 429 heartbeater.cc:347] Connected to a master server at 127.13.148.190:44301
I20260806 04:56:05.117548 429 heartbeater.cc:464] Registering TS with master...
I20260806 04:56:05.118331 429 heartbeater.cc:511] Master 127.13.148.190:44301 requested a full tablet report, sending...
I20260806 04:56:05.119668 32568 ts_manager.cc:194] Registered new tserver with Master: 412cf425291a4bb0a4afa636dd40ccb3 (127.13.148.130:40699)
I20260806 04:56:05.120402 32568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:47549
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
I20260806 04:56:05.158317 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44301
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:43093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:05.458760 434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:05.459056 434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:05.459223 434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:05.465386 434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:05.465516 434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:05.465718 434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:56:05.474081 434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:43093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:44301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:05.475121 434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:05.476279 434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:05.488381 439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:05.488469 440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:05.488418 442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:05.489225 434 server_base.cc:1034] running on GCE node
I20260806 04:56:05.489787 434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:05.490648 434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:05.491868 434 hybrid_clock.cc:648] HybridClock initialized: now 1785992165491812 us; error 66 us; skew 500 ppm
I20260806 04:56:05.492157 434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:05 dist-test-slave-1x4g krb5kdc[32522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992165, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:05.495074 434 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:56:05.496332 434 webserver.cc:460] Webserver started at http://127.13.148.131:45661/ using document root <none> and password file <none>
I20260806 04:56:05.497061 434 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:05.497206 434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:05.497507 434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:05.500133 434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "1411865be3274cffb4d4889375792394"
format_stamp: "Formatted at 2026-08-06 04:56:05 on dist-test-slave-1x4g"
server_key: "03be397d9c10ca799150f364969b7104"
server_key_iv: "37a73252f22e6f98bc9f9c973ef2a1f1"
server_key_version: "encryptionkey@0"
I20260806 04:56:05.500855 434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "1411865be3274cffb4d4889375792394"
format_stamp: "Formatted at 2026-08-06 04:56:05 on dist-test-slave-1x4g"
server_key: "03be397d9c10ca799150f364969b7104"
server_key_iv: "37a73252f22e6f98bc9f9c973ef2a1f1"
server_key_version: "encryptionkey@0"
I20260806 04:56:05.505127 434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:56:05.508260 450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:05.509547 434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:56:05.509711 434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "1411865be3274cffb4d4889375792394"
format_stamp: "Formatted at 2026-08-06 04:56:05 on dist-test-slave-1x4g"
server_key: "03be397d9c10ca799150f364969b7104"
server_key_iv: "37a73252f22e6f98bc9f9c973ef2a1f1"
server_key_version: "encryptionkey@0"
I20260806 04:56:05.509891 434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:05.522025 434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:05.525785 434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:05.526063 434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:05.526856 434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:56:05.528005 434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:56:05.528110 434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:05.528256 434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:56:05.528334 434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:05.558223 434 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:37233
I20260806 04:56:05.558305 562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:37233 every 8 connection(s)
I20260806 04:56:05.559520 434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:56:05.567966 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 434
I20260806 04:56:05.568163 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:56:05.568606 13906 external_mini_cluster.cc:1428] Setting key 29941357b63ae053bb7ad94ebcb15b2e
Aug 06 04:56:05 dist-test-slave-1x4g krb5kdc[32522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992165, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:56:05.576478 32764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.561810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:37107 (local address 127.13.148.190:44301)
0806 04:56:05.561981 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:05.561986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:05.562874 (+ 888us) server_negotiation.cc:407] Connection header received
0806 04:56:05.564364 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:05.564368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:05.564432 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:05.564508 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:05.565835 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.566492 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.567817 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.568378 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.568520 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.571916 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:05.571920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:05.571923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:05.571956 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:05.574210 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:05.574794 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:05.574797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:05.574799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:05.574911 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:05.575257 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:05.575259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:05.575261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:05.575514 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:05.575613 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:05.576111 (+ 498us) server_negotiation.cc:299] Negotiation successful
0806 04:56:05.576265 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:56:05.577448 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.562117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:44301 (local address 127.13.148.131:37107)
0806 04:56:05.562727 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:05.562760 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:05.564136 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:05.564691 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:05.564698 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:05.565177 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:05.565693 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.565709 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.566610 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.566613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.567678 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.567688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.568607 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.568617 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.568933 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.569539 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:05.569566 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:05.571762 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:05.574334 (+ 2572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:05.574339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:05.574349 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:05.574702 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:05.575005 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:05.575008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:05.575017 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:05.575173 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:05.575607 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:05.575611 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:05.575858 (+ 247us) client_negotiation.cc:769] Sending connection context
0806 04:56:05.576053 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 04:56:05.576352 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":236,"threads_started":1}
I20260806 04:56:05.578881 563 heartbeater.cc:347] Connected to a master server at 127.13.148.190:44301
I20260806 04:56:05.579190 563 heartbeater.cc:464] Registering TS with master...
I20260806 04:56:05.579802 563 heartbeater.cc:511] Master 127.13.148.190:44301 requested a full tablet report, sending...
I20260806 04:56:05.581094 32568 ts_manager.cc:194] Registered new tserver with Master: 1411865be3274cffb4d4889375792394 (127.13.148.131:37233)
I20260806 04:56:05.581856 32568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:37107
I20260806 04:56:05.583586 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:56:05.599200 32764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.587379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34464 (local address 127.13.148.190:44301)
0806 04:56:05.587522 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:05.587526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:05.587990 (+ 464us) server_negotiation.cc:407] Connection header received
0806 04:56:05.588114 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:05.588117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:05.588170 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:05.588244 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:05.589681 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.590346 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.591566 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.591905 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.591985 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.594395 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:05.594399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:05.594401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:05.594427 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:05.596819 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:05.597550 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:05.597553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:05.597556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:05.597688 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:05.598076 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:05.598080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:05.598082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:05.598342 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:05.598547 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:05.598867 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 04:56:05.598993 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:56:05.602535 32567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:56:05.608924 32567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:56:05.641168 576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.620266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:37233 (local address 127.0.0.1:44452)
0806 04:56:05.629597 (+ 9331us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:05.629671 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:05.629876 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:05.634220 (+ 4344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:05.634224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:05.634247 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:05.634638 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.634651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.636331 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.636342 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.638577 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.638601 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.640334 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.640339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.640502 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.640552 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 04:56:05.640663 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 04:56:05.640843 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":668,"threads_started":1}
I20260806 04:56:05.641201 578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.620481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44452 (local address 127.13.148.131:37233)
0806 04:56:05.633415 (+ 12934us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:05.633426 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:05.633461 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:56:05.633557 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:05.633575 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:05.633811 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:05.634205 (+ 394us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:05.634840 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.636168 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.638762 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.640157 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.640336 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.640483 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:05.640723 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 04:56:05.640899 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12728,"thread_start_us":13472,"threads_started":1}
I20260806 04:56:05.641710 577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.620390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:46637 (local address 127.0.0.1:54320)
0806 04:56:05.631217 (+ 10827us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:05.631235 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:05.631353 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:05.633858 (+ 2505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:05.633864 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:05.633890 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:05.634355 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.634367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.636094 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.636100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.638629 (+ 2529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.638641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.640951 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.640957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.641129 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.641159 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 04:56:05.641247 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 04:56:05.641350 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10700,"mutex_wait_us":25,"thread_start_us":10591,"threads_started":1}
I20260806 04:56:05.642390 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.620782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54320 (local address 127.13.148.129:46637)
0806 04:56:05.633409 (+ 12627us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:05.633415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:05.633437 (+ 22us) server_negotiation.cc:407] Connection header received
0806 04:56:05.633501 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:05.633506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:05.633677 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:05.633851 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:05.634554 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.635945 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.638846 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.640166 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.640336 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.640483 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:05.641977 (+ 1494us) server_negotiation.cc:299] Negotiation successful
0806 04:56:05.642124 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12475,"thread_start_us":12350,"threads_started":1}
I20260806 04:56:05.642462 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.622143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:40699 (local address 127.0.0.1:56528)
0806 04:56:05.635377 (+ 13234us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:05.635391 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:05.635539 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:05.635975 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:05.635978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:05.636000 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:05.636504 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.636520 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.638476 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.638482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.640012 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.640023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.641813 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.641818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.641970 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.642001 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 04:56:05.642090 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 04:56:05.642193 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13119,"thread_start_us":13006,"threads_started":1}
I20260806 04:56:05.645210 581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.622729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56528 (local address 127.13.148.130:40699)
0806 04:56:05.634979 (+ 12250us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:05.634988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:05.635510 (+ 522us) server_negotiation.cc:407] Connection header received
0806 04:56:05.635669 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:05.635674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:05.635797 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:05.635982 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:05.636734 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.638320 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.640241 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.641646 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.641794 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.641936 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:05.642147 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 04:56:05.642288 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12094,"thread_start_us":12007,"threads_started":1}
I20260806 04:56:05.644757 498 tablet_service.cc:1467] Processing CreateTablet for tablet 7a59284e2b654cd7aac1898b87eee9ec (DEFAULT_TABLE table=test-table [id=effd9168ab8147e386eec16fdce59596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:56:05.645437 32696 tablet_service.cc:1467] Processing CreateTablet for tablet 7a59284e2b654cd7aac1898b87eee9ec (DEFAULT_TABLE table=test-table [id=effd9168ab8147e386eec16fdce59596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:56:05.646011 498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a59284e2b654cd7aac1898b87eee9ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:05.646476 32696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a59284e2b654cd7aac1898b87eee9ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:05.651352 32762 heartbeater.cc:503] Master 127.13.148.190:44301 was elected leader, sending a full tablet report...
I20260806 04:56:05.651722 364 tablet_service.cc:1467] Processing CreateTablet for tablet 7a59284e2b654cd7aac1898b87eee9ec (DEFAULT_TABLE table=test-table [id=effd9168ab8147e386eec16fdce59596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:56:05.652936 364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a59284e2b654cd7aac1898b87eee9ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:05.656658 582 tablet_bootstrap.cc:492] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394: Bootstrap starting.
I20260806 04:56:05.658557 583 tablet_bootstrap.cc:492] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3: Bootstrap starting.
I20260806 04:56:05.661015 582 tablet_bootstrap.cc:654] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:05.661711 583 tablet_bootstrap.cc:654] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:05.662144 582 log.cc:824] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:05.662801 583 log.cc:824] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:05.664523 586 tablet_bootstrap.cc:492] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3: Bootstrap starting.
I20260806 04:56:05.665278 582 tablet_bootstrap.cc:492] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394: No bootstrap required, opened a new log
I20260806 04:56:05.665531 582 ts_tablet_manager.cc:1397] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260806 04:56:05.667200 586 tablet_bootstrap.cc:654] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:05.667529 583 tablet_bootstrap.cc:492] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3: No bootstrap required, opened a new log
I20260806 04:56:05.667757 583 ts_tablet_manager.cc:1397] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260806 04:56:05.668005 586 log.cc:824] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:05.670735 586 tablet_bootstrap.cc:492] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3: No bootstrap required, opened a new log
I20260806 04:56:05.670890 586 ts_tablet_manager.cc:1397] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 04:56:05.674804 582 raft_consensus.cc:348] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1411865be3274cffb4d4889375792394" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37233 } } peers { permanent_uuid: "f8428e53406e4646abc37c0ee38a69a3" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 46637 } } peers { permanent_uuid: "412cf425291a4bb0a4afa636dd40ccb3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40699 } }
I20260806 04:56:05.675194 582 raft_consensus.cc:374] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:05.675300 582 raft_consensus.cc:729] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1411865be3274cffb4d4889375792394, State: Initialized, Role: FOLLOWER
I20260806 04:56:05.675838 582 consensus_queue.cc:260] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1411865be3274cffb4d4889375792394" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37233 } } peers { permanent_uuid: "f8428e53406e4646abc37c0ee38a69a3" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 46637 } } peers { permanent_uuid: "412cf425291a4bb0a4afa636dd40ccb3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40699 } }
I20260806 04:56:05.676862 563 heartbeater.cc:503] Master 127.13.148.190:44301 was elected leader, sending a full tablet report...
I20260806 04:56:05.676988 583 raft_consensus.cc:348] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1411865be3274cffb4d4889375792394" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37233 } } peers { permanent_uuid: "f8428e53406e4646abc37c0ee38a69a3" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 46637 } } peers { permanent_uuid: "412cf425291a4bb0a4afa636dd40ccb3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40699 } }
I20260806 04:56:05.677416 583 raft_consensus.cc:374] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:05.677528 583 raft_consensus.cc:729] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8428e53406e4646abc37c0ee38a69a3, State: Initialized, Role: FOLLOWER
I20260806 04:56:05.677279 586 raft_consensus.cc:348] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1411865be3274cffb4d4889375792394" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37233 } } peers { permanent_uuid: "f8428e53406e4646abc37c0ee38a69a3" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 46637 } } peers { permanent_uuid: "412cf425291a4bb0a4afa636dd40ccb3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40699 } }
I20260806 04:56:05.677711 586 raft_consensus.cc:374] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:05.677855 586 raft_consensus.cc:729] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412cf425291a4bb0a4afa636dd40ccb3, State: Initialized, Role: FOLLOWER
I20260806 04:56:05.678093 583 consensus_queue.cc:260] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1411865be3274cffb4d4889375792394" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37233 } } peers { permanent_uuid: "f8428e53406e4646abc37c0ee38a69a3" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 46637 } } peers { permanent_uuid: "412cf425291a4bb0a4afa636dd40ccb3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40699 } }
I20260806 04:56:05.678555 586 consensus_queue.cc:260] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1411865be3274cffb4d4889375792394" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37233 } } peers { permanent_uuid: "f8428e53406e4646abc37c0ee38a69a3" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 46637 } } peers { permanent_uuid: "412cf425291a4bb0a4afa636dd40ccb3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40699 } }
I20260806 04:56:05.679039 582 ts_tablet_manager.cc:1428] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260806 04:56:05.680032 429 heartbeater.cc:503] Master 127.13.148.190:44301 was elected leader, sending a full tablet report...
I20260806 04:56:05.680481 586 ts_tablet_manager.cc:1428] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260806 04:56:05.682476 583 ts_tablet_manager.cc:1428] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260806 04:56:05.786314 588 raft_consensus.cc:482] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:56:05.786602 588 raft_consensus.cc:504] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1411865be3274cffb4d4889375792394" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37233 } } peers { permanent_uuid: "f8428e53406e4646abc37c0ee38a69a3" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 46637 } } peers { permanent_uuid: "412cf425291a4bb0a4afa636dd40ccb3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40699 } }
I20260806 04:56:05.788285 588 leader_election.cc:288] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8428e53406e4646abc37c0ee38a69a3 (127.13.148.129:46637), 412cf425291a4bb0a4afa636dd40ccb3 (127.13.148.130:40699)
I20260806 04:56:05.793946 590 raft_consensus.cc:482] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:56:05.794224 590 raft_consensus.cc:504] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1411865be3274cffb4d4889375792394" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37233 } } peers { permanent_uuid: "f8428e53406e4646abc37c0ee38a69a3" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 46637 } } peers { permanent_uuid: "412cf425291a4bb0a4afa636dd40ccb3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40699 } }
I20260806 04:56:05.794847 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.788547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:40699 (local address 127.13.148.131:42729)
0806 04:56:05.788884 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:05.788905 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:05.789042 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:05.789495 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:05.789502 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:05.789537 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:05.789808 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.789818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.790954 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.790958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.792758 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.792798 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.794079 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.794086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.794259 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.794291 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 04:56:05.794393 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 04:56:05.794529 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":22,"spinlock_wait_cycles":36224}
I20260806 04:56:05.794862 581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.788958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:42729 (local address 127.13.148.130:40699)
0806 04:56:05.789167 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:05.789172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:05.789192 (+ 20us) server_negotiation.cc:407] Connection header received
0806 04:56:05.789269 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:05.789276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:05.789354 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:05.789448 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:05.789973 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.790832 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.792908 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.793928 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.794074 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.794139 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:05.794480 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 04:56:05.794589 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 04:56:05.795058 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.788678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:36737 (local address 127.13.148.129:46637)
0806 04:56:05.788914 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:05.788922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:05.789613 (+ 691us) server_negotiation.cc:407] Connection header received
0806 04:56:05.789762 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:05.789766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:05.789819 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:05.789935 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:05.790389 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.791186 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.793178 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.794208 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.794336 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.794398 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:05.794708 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 04:56:05.794837 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 04:56:05.795703 591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.788654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:46637 (local address 127.13.148.131:36737)
0806 04:56:05.789494 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:05.789511 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:05.789641 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:05.790005 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:05.790007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:05.790025 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:05.790257 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.790263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.791311 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.791316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.793055 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.793064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.794379 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.794384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.794545 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.794573 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 04:56:05.795233 (+ 660us) client_negotiation.cc:240] Negotiation successful
0806 04:56:05.795319 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":5,"thread_start_us":455,"threads_started":1}
I20260806 04:56:05.795830 384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a59284e2b654cd7aac1898b87eee9ec" candidate_uuid: "1411865be3274cffb4d4889375792394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412cf425291a4bb0a4afa636dd40ccb3" is_pre_election: true
I20260806 04:56:05.796375 384 raft_consensus.cc:2454] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1411865be3274cffb4d4889375792394 in term 0.
I20260806 04:56:05.797128 590 leader_election.cc:288] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1411865be3274cffb4d4889375792394 (127.13.148.131:37233), f8428e53406e4646abc37c0ee38a69a3 (127.13.148.129:46637)
I20260806 04:56:05.797206 32716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a59284e2b654cd7aac1898b87eee9ec" candidate_uuid: "1411865be3274cffb4d4889375792394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8428e53406e4646abc37c0ee38a69a3" is_pre_election: true
I20260806 04:56:05.797884 32716 raft_consensus.cc:2454] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1411865be3274cffb4d4889375792394 in term 0.
I20260806 04:56:05.798450 452 leader_election.cc:302] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1411865be3274cffb4d4889375792394, 412cf425291a4bb0a4afa636dd40ccb3; no voters:
I20260806 04:56:05.798889 588 raft_consensus.cc:2781] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:56:05.799130 588 raft_consensus.cc:482] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:56:05.799249 588 raft_consensus.cc:3037] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:05.801859 588 raft_consensus.cc:504] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1411865be3274cffb4d4889375792394" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37233 } } peers { permanent_uuid: "f8428e53406e4646abc37c0ee38a69a3" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 46637 } } peers { permanent_uuid: "412cf425291a4bb0a4afa636dd40ccb3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40699 } }
I20260806 04:56:05.802418 588 leader_election.cc:288] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [CANDIDATE]: Term 1 election: Requested vote from peers f8428e53406e4646abc37c0ee38a69a3 (127.13.148.129:46637), 412cf425291a4bb0a4afa636dd40ccb3 (127.13.148.130:40699)
I20260806 04:56:05.802834 32716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a59284e2b654cd7aac1898b87eee9ec" candidate_uuid: "1411865be3274cffb4d4889375792394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8428e53406e4646abc37c0ee38a69a3"
I20260806 04:56:05.803030 32716 raft_consensus.cc:3037] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:05.803093 384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a59284e2b654cd7aac1898b87eee9ec" candidate_uuid: "1411865be3274cffb4d4889375792394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412cf425291a4bb0a4afa636dd40ccb3"
I20260806 04:56:05.803272 384 raft_consensus.cc:3037] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:05.803841 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.797008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:60719 (local address 127.13.148.129:46637)
0806 04:56:05.797494 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:05.797501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:05.797920 (+ 419us) server_negotiation.cc:407] Connection header received
0806 04:56:05.798079 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:05.798086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:05.798154 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:05.798261 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:05.799025 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.799793 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.801751 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.802723 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.802917 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.802974 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:05.803503 (+ 529us) server_negotiation.cc:299] Negotiation successful
0806 04:56:05.803618 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343}
I20260806 04:56:05.804586 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.796167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:46637 (local address 127.13.148.130:60719)
0806 04:56:05.797759 (+ 1592us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:05.797778 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:05.797928 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:05.798419 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:05.798427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:05.798456 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:05.798851 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.798864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.799948 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.799953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.801609 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.801620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.803150 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.803156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.803323 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.803360 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 04:56:05.803977 (+ 617us) client_negotiation.cc:240] Negotiation successful
0806 04:56:05.804163 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1409,"thread_start_us":1615,"threads_started":1}
I20260806 04:56:05.805223 578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.796471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:38357 (local address 127.13.148.131:37233)
0806 04:56:05.796694 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:05.796700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:05.798673 (+ 1973us) server_negotiation.cc:407] Connection header received
0806 04:56:05.798953 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:05.798958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:05.799030 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:05.799100 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:05.799816 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.800906 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.803065 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.804148 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.804259 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.804321 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:05.804894 (+ 573us) server_negotiation.cc:299] Negotiation successful
0806 04:56:05.805009 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:56:05.805318 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.796243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:37233 (local address 127.13.148.130:38357)
0806 04:56:05.798575 (+ 2332us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:05.798595 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:05.798772 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:05.799288 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:05.799292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:05.799317 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:05.799663 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.799674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.801037 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.801042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.802909 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:05.802921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.804533 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.804546 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:05.804712 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.804741 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 04:56:05.804837 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 04:56:05.804957 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2211,"thread_start_us":1627,"threads_started":1}
I20260806 04:56:05.806222 384 raft_consensus.cc:2454] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1411865be3274cffb4d4889375792394 in term 1.
I20260806 04:56:05.806094 518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a59284e2b654cd7aac1898b87eee9ec" candidate_uuid: "412cf425291a4bb0a4afa636dd40ccb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1411865be3274cffb4d4889375792394" is_pre_election: true
I20260806 04:56:05.806352 32716 raft_consensus.cc:2454] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1411865be3274cffb4d4889375792394 in term 1.
I20260806 04:56:05.806541 518 raft_consensus.cc:2379] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 412cf425291a4bb0a4afa636dd40ccb3 in current term 1: Already voted for candidate 1411865be3274cffb4d4889375792394 in this term.
I20260806 04:56:05.806790 451 leader_election.cc:302] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1411865be3274cffb4d4889375792394, f8428e53406e4646abc37c0ee38a69a3; no voters:
I20260806 04:56:05.807116 32716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a59284e2b654cd7aac1898b87eee9ec" candidate_uuid: "412cf425291a4bb0a4afa636dd40ccb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8428e53406e4646abc37c0ee38a69a3" is_pre_election: true
I20260806 04:56:05.807181 588 raft_consensus.cc:2781] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:05.807415 32716 raft_consensus.cc:2379] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 412cf425291a4bb0a4afa636dd40ccb3 in current term 1: Already voted for candidate 1411865be3274cffb4d4889375792394 in this term.
I20260806 04:56:05.807715 588 raft_consensus.cc:686] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [term 1 LEADER]: Becoming Leader. State: Replica: 1411865be3274cffb4d4889375792394, State: Running, Role: LEADER
I20260806 04:56:05.808043 316 leader_election.cc:302] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 412cf425291a4bb0a4afa636dd40ccb3; no voters: 1411865be3274cffb4d4889375792394, f8428e53406e4646abc37c0ee38a69a3
I20260806 04:56:05.808440 590 raft_consensus.cc:2726] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:56:05.808584 588 consensus_queue.cc:237] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1411865be3274cffb4d4889375792394" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37233 } } peers { permanent_uuid: "f8428e53406e4646abc37c0ee38a69a3" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 46637 } } peers { permanent_uuid: "412cf425291a4bb0a4afa636dd40ccb3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40699 } }
W20260806 04:56:05.810882 564 tablet.cc:2369] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:56:05.813036 32566 catalog_manager.cc:5495] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1411865be3274cffb4d4889375792394 (127.13.148.131). New cstate: current_term: 1 leader_uuid: "1411865be3274cffb4d4889375792394" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1411865be3274cffb4d4889375792394" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 37233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8428e53406e4646abc37c0ee38a69a3" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 46637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "412cf425291a4bb0a4afa636dd40ccb3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40699 } health_report { overall_health: UNKNOWN } } }
W20260806 04:56:05.847844 430 tablet.cc:2369] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:56:05.865615 578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.860914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44460 (local address 127.13.148.131:37233)
0806 04:56:05.861118 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:05.861123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:05.861146 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:56:05.861317 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:05.861320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:05.861384 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:05.861490 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:05.861792 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.862431 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.863608 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.863959 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.864045 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.864504 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:05.865052 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:05.865169 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:05.865277 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 04:56:05.865308 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 04:56:05.872613 32716 raft_consensus.cc:1261] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3 [term 1 FOLLOWER]: Refusing update from remote peer 1411865be3274cffb4d4889375792394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:56:05.872669 384 raft_consensus.cc:1261] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3 [term 1 FOLLOWER]: Refusing update from remote peer 1411865be3274cffb4d4889375792394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:56:05.873481 594 consensus_queue.cc:1035] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8428e53406e4646abc37c0ee38a69a3" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 46637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:56:05.873843 588 consensus_queue.cc:1035] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "412cf425291a4bb0a4afa636dd40ccb3" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 04:56:05.875296 32763 tablet.cc:2369] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:56:05.886795 597 mvcc.cc:205] Tried to move back new op lower bound from 7315423911405805568 to 7315423911156293632. Current Snapshot: MvccSnapshot[applied={T|T < 7315423911405805568}]
I20260806 04:56:05.899441 32565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:56:05.899791 32565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:56:05.903308 32565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6df5e4e8ba844a2ca935f4fb07cd435a: Sending DeleteTablet for 3 replicas of tablet 7a59284e2b654cd7aac1898b87eee9ec
I20260806 04:56:05.904284 32696 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a59284e2b654cd7aac1898b87eee9ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:56:05 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:54320
I20260806 04:56:05.904289 498 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a59284e2b654cd7aac1898b87eee9ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:56:05 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:44452
I20260806 04:56:05.904383 364 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a59284e2b654cd7aac1898b87eee9ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:56:05 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:56528
I20260806 04:56:05.908221 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 32631
I20260806 04:56:05.917593 609 tablet_replica.cc:330] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3: stopping tablet replica
I20260806 04:56:05.918076 609 raft_consensus.cc:2229] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:56:05.918525 609 raft_consensus.cc:2258] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:56:05.920344 609 ts_tablet_manager.cc:1905] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:56:05.923166 607 tablet_replica.cc:330] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394: stopping tablet replica
I20260806 04:56:05.923556 607 raft_consensus.cc:2229] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:56:05.924265 607 raft_consensus.cc:2258] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:56:05.926321 607 ts_tablet_manager.cc:1905] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:56:05.927001 609 ts_tablet_manager.cc:1918] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:56:05.927213 609 log.cc:1196] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/7a59284e2b654cd7aac1898b87eee9ec
Aug 06 04:56:05 dist-test-slave-1x4g krb5kdc[32522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:56:05.927795 609 ts_tablet_manager.cc:1939] T 7a59284e2b654cd7aac1898b87eee9ec P 412cf425291a4bb0a4afa636dd40ccb3: Deleting consensus metadata
I20260806 04:56:05.929435 608 tablet_replica.cc:330] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3: stopping tablet replica
I20260806 04:56:05.929919 32553 catalog_manager.cc:4841] TS 412cf425291a4bb0a4afa636dd40ccb3 (127.13.148.130:40699): tablet 7a59284e2b654cd7aac1898b87eee9ec (table test-table [id=effd9168ab8147e386eec16fdce59596]) successfully deleted
I20260806 04:56:05.932403 608 raft_consensus.cc:2229] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:56:05.932901 608 raft_consensus.cc:2258] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:56:05.933143 607 ts_tablet_manager.cc:1918] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:56:05.933315 607 log.cc:1196] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/7a59284e2b654cd7aac1898b87eee9ec
I20260806 04:56:05.933833 607 ts_tablet_manager.cc:1939] T 7a59284e2b654cd7aac1898b87eee9ec P 1411865be3274cffb4d4889375792394: Deleting consensus metadata
I20260806 04:56:05.934899 608 ts_tablet_manager.cc:1905] T 7a59284e2b654cd7aac1898b87eee9ec P f8428e53406e4646abc37c0ee38a69a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:56:05.935365 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:05.909207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54326 (local address 127.13.148.129:46637)
0806 04:56:05.909392 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:05.909399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:05.913243 (+ 3844us) server_negotiation.cc:407] Connection header received
0806 04:56:05.913679 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:05.913686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:05.913843 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:05.913930 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:05.917753 (+ 3823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.918870 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.922762 (+ 3892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:05.923322 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:05.923427 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:05.928605 (+ 5178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:05.928620 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:05.928636 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:05.928681 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:05.932218 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:05.933004 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:05.933029 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:05.933033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:05.933173 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:05.933607 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:05.933613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:05.933615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:05.934056 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:05.934309 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:05.934850 (+ 541us) server_negotiation.cc:299] Negotiation successful
0806 04:56:05.935058 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:56:05.935330 32554 catalog_manager.cc:4841] TS 1411865be3274cffb4d4889375792394 (127.13.148.131:37233): tablet 7a59284e2b654cd7aac1898b87eee9ec (table test-table [id=effd9168ab8147e386eec16fdce59596]) successfully deleted
I20260806 04:56:05.936652 32757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:06.017334 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 32631
I20260806 04:56:06.036850 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 300
W20260806 04:56:06.037084 32552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.129:46637 (error 108)
I20260806 04:56:06.039418 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:06.038903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:46637 (local address 127.0.0.1:54328)
0806 04:56:06.039101 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:06.039227 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:46637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 04:56:06.039714 32552 catalog_manager.cc:4584] TS f8428e53406e4646abc37c0ee38a69a3 (127.13.148.129:46637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7a59284e2b654cd7aac1898b87eee9ec: Network error: Client connection negotiation failed: client connection to 127.13.148.129:46637: connect: Connection refused (error 111)
Aug 06 04:56:06 dist-test-slave-1x4g krb5kdc[32522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:56:06.052918 581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:06.037685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.13.148.130:40699)
0806 04:56:06.037868 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:06.037874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:06.037897 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:56:06.038137 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:06.038144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:06.038233 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:06.038318 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:06.040256 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:06.040987 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:06.042294 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:06.042954 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:06.043117 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:06.047082 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:06.047094 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:06.047106 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:06.047132 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:06.050210 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:06.050882 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:06.050889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:06.050891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:06.051015 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:06.051380 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:06.051385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:06.051390 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:06.051726 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:06.052017 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:06.052308 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 04:56:06.052548 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 04:56:06.053578 424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:06.093422 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:06.092889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:46637 (local address 127.0.0.1:54340)
0806 04:56:06.093134 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:06.093237 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:46637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 04:56:06.094537 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:06.094158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:46637 (local address 127.0.0.1:54342)
0806 04:56:06.094320 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:06.094402 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:46637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 04:56:06.139367 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 300
I20260806 04:56:06.155364 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:06.154809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:46637 (local address 127.0.0.1:54354)
0806 04:56:06.155058 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:06.155176 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:46637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 04:56:06.156718 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:06.156218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:46637 (local address 127.0.0.1:54358)
0806 04:56:06.156430 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:06.156531 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:46637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 04:56:06.158974 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 434
Aug 06 04:56:06 dist-test-slave-1x4g krb5kdc[32522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
I20260806 04:56:06.176981 578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:06.159729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.13.148.131:37233)
0806 04:56:06.159909 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:06.159913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:06.159930 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:56:06.160146 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:06.160150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:06.160218 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:06.160296 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:06.162062 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:06.162825 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:06.164672 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:06.165311 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:06.165407 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:06.170775 (+ 5368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:06.170786 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:06.170805 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:06.170847 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:06.174137 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:06.174874 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:06.174882 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:06.174885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:06.175051 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:06.175419 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:06.175424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:06.175428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:06.175782 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:06.176104 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:06.176338 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 04:56:06.176559 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 04:56:06.177649 558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:06.237442 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:06.236961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:46637 (local address 127.0.0.1:54368)
0806 04:56:06.237172 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:06.237262 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:46637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 04:56:06.238543 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:06.238129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:46637 (local address 127.0.0.1:54380)
0806 04:56:06.238336 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:06.238413 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:46637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260806 04:56:06.295893 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 434
I20260806 04:56:06.319393 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 32535
I20260806 04:56:06.320492 32599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:06.430368 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 32535
I20260806 04:56:06.430413 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:06.429899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:46637 (local address 127.0.0.1:54388)
0806 04:56:06.430154 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:06.430254 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:46637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
2026-08-06T04:56:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4074 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T04:56:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:06Z Disabled control of system clock
I20260806 04:56:06.507814 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:33201
--webserver_interface=127.13.148.190
--webserver_port=0
--builtin_ntp_servers=127.13.148.148:38553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:33201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:56:06.815259 620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:06.815493 620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:06.815572 620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:06.822233 620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:06.822324 620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:06.822361 620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:06.822422 620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:06.822465 620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:06.832445 620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:38553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:33201
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:33201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:06.833736 620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:06.835333 620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:06.846648 628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:06.846730 625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:06.846704 626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:06.847484 620 server_base.cc:1034] running on GCE node
I20260806 04:56:06.848202 620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:06.849551 620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:06.850732 620 hybrid_clock.cc:648] HybridClock initialized: now 1785992166850683 us; error 44 us; skew 500 ppm
I20260806 04:56:06.851040 620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:56:06.853332 620 webserver.cc:460] Webserver started at http://127.13.148.190:35183/ using document root <none> and password file <none>
I20260806 04:56:06.853945 620 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:06.854048 620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:06.854344 620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:06.857065 620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "cbc00cf3f33140dba6821291803420d6"
format_stamp: "Formatted at 2026-08-06 04:56:06 on dist-test-slave-1x4g"
server_key: "cda47f9d19a1e14bba24da5e834f6b00"
server_key_iv: "d55ae621a7e155e64c1001c7d44348a3"
server_key_version: "encryptionkey@0"
I20260806 04:56:06.857851 620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "cbc00cf3f33140dba6821291803420d6"
format_stamp: "Formatted at 2026-08-06 04:56:06 on dist-test-slave-1x4g"
server_key: "cda47f9d19a1e14bba24da5e834f6b00"
server_key_iv: "d55ae621a7e155e64c1001c7d44348a3"
server_key_version: "encryptionkey@0"
I20260806 04:56:06.862322 620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:56:06.865386 634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:06.866557 620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:56:06.866720 620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "cbc00cf3f33140dba6821291803420d6"
format_stamp: "Formatted at 2026-08-06 04:56:06 on dist-test-slave-1x4g"
server_key: "cda47f9d19a1e14bba24da5e834f6b00"
server_key_iv: "d55ae621a7e155e64c1001c7d44348a3"
server_key_version: "encryptionkey@0"
I20260806 04:56:06.866955 620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:06.895977 620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:06.896763 620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:06.897094 620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:06.914176 620 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:33201
I20260806 04:56:06.914196 685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:33201 every 8 connection(s)
I20260806 04:56:06.915458 620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:06.917774 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 620
I20260806 04:56:06.917963 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:06.918444 13906 external_mini_cluster.cc:1428] Setting key e78e55b7338bcb61900ef074a965412a
I20260806 04:56:06.919327 686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:06.931083 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:06.920217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.13.148.190:33201)
0806 04:56:06.921119 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:06.921128 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:06.921163 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:56:06.922456 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:06.922475 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:06.922880 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:06.923198 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:06.923806 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:06.925070 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:06.926801 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:06.927393 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:06.927519 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:06.928862 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:06.928871 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:06.928883 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:06.929449 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:06.929517 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:06.929553 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:06.929636 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:06.929857 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 04:56:06.930207 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":336,"threads_started":1}
I20260806 04:56:06.932395 686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6: Bootstrap starting.
I20260806 04:56:06.935529 686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:06.936553 686 log.cc:824] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:06.939384 686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6: No bootstrap required, opened a new log
I20260806 04:56:06.945874 686 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc00cf3f33140dba6821291803420d6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33201 } }
I20260806 04:56:06.946195 686 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:06.946321 686 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbc00cf3f33140dba6821291803420d6, State: Initialized, Role: FOLLOWER
I20260806 04:56:06.946832 686 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc00cf3f33140dba6821291803420d6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33201 } }
I20260806 04:56:06.947103 686 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:06.947294 686 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:06.947470 686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:06.949815 686 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc00cf3f33140dba6821291803420d6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33201 } }
I20260806 04:56:06.950237 686 leader_election.cc:302] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbc00cf3f33140dba6821291803420d6; no voters:
I20260806 04:56:06.950750 686 leader_election.cc:288] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:06.950860 691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:06.951777 691 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [term 1 LEADER]: Becoming Leader. State: Replica: cbc00cf3f33140dba6821291803420d6, State: Running, Role: LEADER
I20260806 04:56:06.952270 691 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc00cf3f33140dba6821291803420d6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33201 } }
I20260806 04:56:06.952605 686 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:06.954922 693 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc00cf3f33140dba6821291803420d6. Latest consensus state: current_term: 1 leader_uuid: "cbc00cf3f33140dba6821291803420d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc00cf3f33140dba6821291803420d6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33201 } } }
I20260806 04:56:06.955011 692 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbc00cf3f33140dba6821291803420d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc00cf3f33140dba6821291803420d6" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33201 } } }
I20260806 04:56:06.955327 693 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:06.955389 692 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:06.957093 697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:06.963330 697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:06.970834 697 catalog_manager.cc:1347] Generated new cluster ID: 83e931daac534017974ace3ec704cd71
I20260806 04:56:06.970975 697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:06.981102 697 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:06.982551 697 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:06.999725 697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6: Generated new TSK 0
I20260806 04:56:07.000450 697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:07.004652 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33201
--builtin_ntp_servers=127.13.148.148:38553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:56:07.314199 711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:07.314422 711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:07.314495 711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:07.321313 711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:07.321398 711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:07.321491 711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:56:07.329465 711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:38553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:07.330519 711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:07.332063 711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:07.344408 719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:07.344514 716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:07.344494 717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:07.345311 711 server_base.cc:1034] running on GCE node
I20260806 04:56:07.345781 711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:07.346517 711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:07.347704 711 hybrid_clock.cc:648] HybridClock initialized: now 1785992167347680 us; error 36 us; skew 500 ppm
I20260806 04:56:07.347993 711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:56:07.350139 711 webserver.cc:460] Webserver started at http://127.13.148.129:46059/ using document root <none> and password file <none>
I20260806 04:56:07.350790 711 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:07.350899 711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:07.351199 711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:07.353956 711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "bb75984a6d5c42e1b8a1bf90d1f7ca31"
format_stamp: "Formatted at 2026-08-06 04:56:07 on dist-test-slave-1x4g"
server_key: "b83c96907767be85fddbbc273ab93c8f"
server_key_iv: "edd184e2c78660035075092f940e24fe"
server_key_version: "encryptionkey@0"
I20260806 04:56:07.354673 711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "bb75984a6d5c42e1b8a1bf90d1f7ca31"
format_stamp: "Formatted at 2026-08-06 04:56:07 on dist-test-slave-1x4g"
server_key: "b83c96907767be85fddbbc273ab93c8f"
server_key_iv: "edd184e2c78660035075092f940e24fe"
server_key_version: "encryptionkey@0"
I20260806 04:56:07.359575 711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 04:56:07.362597 725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:07.363684 711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:56:07.363842 711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "bb75984a6d5c42e1b8a1bf90d1f7ca31"
format_stamp: "Formatted at 2026-08-06 04:56:07 on dist-test-slave-1x4g"
server_key: "b83c96907767be85fddbbc273ab93c8f"
server_key_iv: "edd184e2c78660035075092f940e24fe"
server_key_version: "encryptionkey@0"
I20260806 04:56:07.364022 711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:07.379230 711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:07.379923 711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:07.380213 711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:07.381078 711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:56:07.382380 711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:56:07.382485 711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:07.382623 711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:56:07.382718 711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:07.413393 711 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:36565
I20260806 04:56:07.413465 840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:36565 every 8 connection(s)
I20260806 04:56:07.414634 711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:56:07.419579 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 711
I20260806 04:56:07.419736 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:56:07.420271 13906 external_mini_cluster.cc:1428] Setting key 9216bcba5d4d94afd7f1960d109316a5
I20260806 04:56:07.423132 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33201
--builtin_ntp_servers=127.13.148.148:38553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:56:07.427750 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:07.416975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:44603 (local address 127.13.148.190:33201)
0806 04:56:07.417159 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:07.417163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:07.418462 (+ 1299us) server_negotiation.cc:407] Connection header received
0806 04:56:07.420546 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:07.420550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:07.420611 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:07.420712 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:07.421929 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:07.422692 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:07.425002 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:07.425422 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:07.425526 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:07.427013 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:07.427018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:07.427021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:07.427041 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:07.427075 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:07.427088 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:07.427161 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:07.427425 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 04:56:07.427508 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:56:07.429143 843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:07.417420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33201 (local address 127.13.148.129:44603)
0806 04:56:07.418302 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:07.418347 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:07.420264 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:07.421003 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:07.421011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:07.421122 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:07.421759 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:07.421775 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:07.422836 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:07.422843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:07.424790 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:07.424824 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:07.425634 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:07.425642 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:07.425924 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:07.426641 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:56:07.426645 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:07.426690 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:56:07.426775 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:07.427189 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:07.427202 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:07.427212 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 04:56:07.427382 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 04:56:07.427777 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":273,"threads_started":1}
I20260806 04:56:07.430588 841 heartbeater.cc:347] Connected to a master server at 127.13.148.190:33201
I20260806 04:56:07.430869 841 heartbeater.cc:464] Registering TS with master...
I20260806 04:56:07.431655 841 heartbeater.cc:511] Master 127.13.148.190:33201 requested a full tablet report, sending...
I20260806 04:56:07.433883 651 ts_manager.cc:194] Registered new tserver with Master: bb75984a6d5c42e1b8a1bf90d1f7ca31 (127.13.148.129:36565)
I20260806 04:56:07.435348 651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.129:44603
W20260806 04:56:07.739626 844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:07.739878 844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:07.740042 844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:07.746495 844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:07.746608 844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:07.746814 844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:56:07.755159 844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:38553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:07.756227 844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:07.758129 844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:07.770618 851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:07.770627 853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:07.771131 850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:07.771587 844 server_base.cc:1034] running on GCE node
I20260806 04:56:07.772223 844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:07.773160 844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:07.774355 844 hybrid_clock.cc:648] HybridClock initialized: now 1785992167774300 us; error 53 us; skew 500 ppm
I20260806 04:56:07.774647 844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:56:07.776768 844 webserver.cc:460] Webserver started at http://127.13.148.130:34057/ using document root <none> and password file <none>
I20260806 04:56:07.777539 844 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:07.777671 844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:07.778039 844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:07.780777 844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "92befa0bd5a340969cb6914889348f5d"
format_stamp: "Formatted at 2026-08-06 04:56:07 on dist-test-slave-1x4g"
server_key: "ffbf3034cd600a1268cd15aaad77fe11"
server_key_iv: "b5e7836c053cbd7107dfcc5a517c831c"
server_key_version: "encryptionkey@0"
I20260806 04:56:07.781589 844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "92befa0bd5a340969cb6914889348f5d"
format_stamp: "Formatted at 2026-08-06 04:56:07 on dist-test-slave-1x4g"
server_key: "ffbf3034cd600a1268cd15aaad77fe11"
server_key_iv: "b5e7836c053cbd7107dfcc5a517c831c"
server_key_version: "encryptionkey@0"
I20260806 04:56:07.786414 844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:56:07.789603 859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:07.790723 844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260806 04:56:07.790940 844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "92befa0bd5a340969cb6914889348f5d"
format_stamp: "Formatted at 2026-08-06 04:56:07 on dist-test-slave-1x4g"
server_key: "ffbf3034cd600a1268cd15aaad77fe11"
server_key_iv: "b5e7836c053cbd7107dfcc5a517c831c"
server_key_version: "encryptionkey@0"
I20260806 04:56:07.791167 844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:07.828940 844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:07.829720 844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:07.830050 844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:07.830969 844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:56:07.832296 844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:56:07.832422 844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:07.832554 844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:56:07.832649 844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:07.865614 844 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:38677
I20260806 04:56:07.865660 974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:38677 every 8 connection(s)
I20260806 04:56:07.867017 844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:56:07.872267 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 844
I20260806 04:56:07.872470 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:56:07.872943 13906 external_mini_cluster.cc:1428] Setting key d5951a1ee74a203842e73f80875dd43b
I20260806 04:56:07.875936 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33201
--builtin_ntp_servers=127.13.148.148:38553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:56:07.880487 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:07.869565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:54853 (local address 127.13.148.190:33201)
0806 04:56:07.869751 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:07.869756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:07.871082 (+ 1326us) server_negotiation.cc:407] Connection header received
0806 04:56:07.872966 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:07.872972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:07.873056 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:07.873159 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:07.874281 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:07.875114 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:07.877710 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:07.878110 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:07.878204 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:07.879816 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:07.879820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:07.879823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:07.879838 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:07.879868 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:07.879878 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:07.879937 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:07.880210 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 04:56:07.880278 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 04:56:07.881880 977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:07.870039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33201 (local address 127.13.148.130:54853)
0806 04:56:07.870933 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:07.870968 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:07.872668 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:07.873361 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:07.873369 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:07.873445 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:07.874102 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:07.874139 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:07.875292 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:07.875297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:07.877553 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:07.877574 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:07.878345 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:07.878354 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:07.878697 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:07.879492 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:56:07.879497 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:07.879544 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:56:07.879632 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:07.879972 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:07.879985 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:07.880012 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 04:56:07.880197 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 04:56:07.880564 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":264,"threads_started":1}
I20260806 04:56:07.883071 975 heartbeater.cc:347] Connected to a master server at 127.13.148.190:33201
I20260806 04:56:07.883337 975 heartbeater.cc:464] Registering TS with master...
I20260806 04:56:07.884037 975 heartbeater.cc:511] Master 127.13.148.190:33201 requested a full tablet report, sending...
I20260806 04:56:07.885617 651 ts_manager.cc:194] Registered new tserver with Master: 92befa0bd5a340969cb6914889348f5d (127.13.148.130:38677)
I20260806 04:56:07.886523 651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.130:54853
W20260806 04:56:08.184454 978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:08.184689 978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:08.184856 978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:08.191282 978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:08.191365 978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:08.191548 978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:56:08.199862 978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:38553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:33201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:08.201015 978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:08.202715 978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:08.214965 984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:08.215047 986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:08.215413 983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:08.215710 978 server_base.cc:1034] running on GCE node
I20260806 04:56:08.216228 978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:08.217191 978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:08.218423 978 hybrid_clock.cc:648] HybridClock initialized: now 1785992168218392 us; error 42 us; skew 500 ppm
I20260806 04:56:08.218729 978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:56:08.220899 978 webserver.cc:460] Webserver started at http://127.13.148.131:34625/ using document root <none> and password file <none>
I20260806 04:56:08.221519 978 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:08.221645 978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:08.221933 978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:08.224531 978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "daeb5a0bcc564851b4b310ed1069158e"
format_stamp: "Formatted at 2026-08-06 04:56:08 on dist-test-slave-1x4g"
server_key: "700633f8da0bd90d8d22223e7b5e6f33"
server_key_iv: "8a138ec489a509c7bb95f03b42783cda"
server_key_version: "encryptionkey@0"
I20260806 04:56:08.225322 978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "daeb5a0bcc564851b4b310ed1069158e"
format_stamp: "Formatted at 2026-08-06 04:56:08 on dist-test-slave-1x4g"
server_key: "700633f8da0bd90d8d22223e7b5e6f33"
server_key_iv: "8a138ec489a509c7bb95f03b42783cda"
server_key_version: "encryptionkey@0"
I20260806 04:56:08.229810 978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:56:08.232874 992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:08.234165 978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:56:08.234282 978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "daeb5a0bcc564851b4b310ed1069158e"
format_stamp: "Formatted at 2026-08-06 04:56:08 on dist-test-slave-1x4g"
server_key: "700633f8da0bd90d8d22223e7b5e6f33"
server_key_iv: "8a138ec489a509c7bb95f03b42783cda"
server_key_version: "encryptionkey@0"
I20260806 04:56:08.234437 978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:08.248152 978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:08.248863 978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:08.249114 978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:08.249905 978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:56:08.251124 978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:56:08.251204 978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:08.251282 978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:56:08.251330 978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:08.280717 978 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:45225
I20260806 04:56:08.280762 1108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:45225 every 8 connection(s)
I20260806 04:56:08.281852 978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:56:08.284127 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 978
I20260806 04:56:08.284315 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:56:08.285000 13906 external_mini_cluster.cc:1428] Setting key 5a2c19d2f021f327a708081451744519
I20260806 04:56:08.293491 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.284040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:35169 (local address 127.13.148.190:33201)
0806 04:56:08.284240 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:08.284246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:08.285903 (+ 1657us) server_negotiation.cc:407] Connection header received
0806 04:56:08.287871 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:08.287875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:08.287933 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:08.288004 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:08.289338 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.289968 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.291272 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.291658 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.291762 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:08.292895 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:08.292899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:08.292902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:08.292918 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:08.292946 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:08.292957 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:08.293037 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:08.293216 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 04:56:08.293283 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 04:56:08.294493 1111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.284617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:33201 (local address 127.13.148.131:35169)
0806 04:56:08.285741 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:08.285795 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:08.287607 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:08.288228 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:08.288238 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:08.288367 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:08.289170 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:08.289188 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.290090 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.290093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:08.291152 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:08.291163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.291842 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.291847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:08.292028 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:08.292585 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:56:08.292587 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:08.292618 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:56:08.292681 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:08.293053 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:08.293061 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:08.293066 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 04:56:08.293190 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 04:56:08.293427 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":362,"threads_started":1}
I20260806 04:56:08.295733 1109 heartbeater.cc:347] Connected to a master server at 127.13.148.190:33201
I20260806 04:56:08.295993 1109 heartbeater.cc:464] Registering TS with master...
I20260806 04:56:08.296696 1109 heartbeater.cc:511] Master 127.13.148.190:33201 requested a full tablet report, sending...
I20260806 04:56:08.298070 651 ts_manager.cc:194] Registered new tserver with Master: daeb5a0bcc564851b4b310ed1069158e (127.13.148.131:45225)
I20260806 04:56:08.298995 651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.131:35169
I20260806 04:56:08.299973 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:56:08.309041 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.303535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.13.148.190:33201)
0806 04:56:08.303665 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:08.303669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:08.304120 (+ 451us) server_negotiation.cc:407] Connection header received
0806 04:56:08.304215 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:08.304218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:08.304276 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:08.304349 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:08.304828 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.305569 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.306899 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.307302 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.307384 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:08.308391 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:08.308395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:08.308398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:08.308413 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:08.308439 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:08.308457 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:08.308559 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:08.308714 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 04:56:08.308754 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:56:08.312071 651 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:56:08.318490 651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:56:08.338707 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.328978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.13.148.130:38677)
0806 04:56:08.329650 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:08.329669 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:08.330250 (+ 581us) server_negotiation.cc:407] Connection header received
0806 04:56:08.330673 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:08.330681 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:08.330901 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:08.331073 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:08.332236 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.333606 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.336260 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.337442 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.337657 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:08.337900 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:08.338139 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 04:56:08.338300 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":395,"threads_started":1}
I20260806 04:56:08.338811 1121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.328552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:38677 (local address 127.0.0.1:50094)
0806 04:56:08.330068 (+ 1516us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:08.330119 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:08.330397 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:08.331455 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:08.331462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:08.331508 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:08.332032 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:08.332045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.333750 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.333757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:08.336052 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:08.336065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.337670 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.337678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:08.337891 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:08.337937 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 04:56:08.338069 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 04:56:08.338413 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1264,"mutex_wait_us":33,"spinlock_wait_cycles":1408,"thread_start_us":1108,"threads_started":1}
I20260806 04:56:08.340015 1124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.329539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58420 (local address 127.13.148.129:36565)
0806 04:56:08.330072 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:08.330080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:08.330599 (+ 519us) server_negotiation.cc:407] Connection header received
0806 04:56:08.330839 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:08.330845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:08.331025 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:08.331160 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:08.332716 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.334169 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.336495 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.337991 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.338160 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:08.338372 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:08.339527 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0806 04:56:08.339679 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":254,"threads_started":1}
I20260806 04:56:08.340648 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.330589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:45225 (local address 127.0.0.1:51472)
0806 04:56:08.332684 (+ 2095us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:08.332699 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:08.332816 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:08.333625 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:08.333629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:08.333651 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:08.334129 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:08.334142 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.335470 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.335475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:08.337729 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:08.337745 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.339886 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.339891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:08.340058 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:08.340089 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 04:56:08.340170 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 04:56:08.340306 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1971,"mutex_wait_us":87,"thread_start_us":1802,"threads_started":1}
I20260806 04:56:08.341040 1122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.328895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36565 (local address 127.0.0.1:58420)
0806 04:56:08.330506 (+ 1611us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:08.330523 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:08.330688 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:08.331463 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:08.331466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:08.331515 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:08.332487 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:08.332501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.334295 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.334304 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:08.336304 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:08.336319 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.338506 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.338513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:08.338684 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:08.338719 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 04:56:08.340584 (+ 1865us) client_negotiation.cc:240] Negotiation successful
0806 04:56:08.340720 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1456,"mutex_wait_us":523,"spinlock_wait_cycles":14976,"thread_start_us":1307,"threads_started":1}
I20260806 04:56:08.341620 1126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.330894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51472 (local address 127.13.148.131:45225)
0806 04:56:08.332205 (+ 1311us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:08.332214 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:08.332920 (+ 706us) server_negotiation.cc:407] Connection header received
0806 04:56:08.333005 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:08.333010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:08.333215 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:08.333392 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:08.334345 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.335297 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.337969 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.339425 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.339605 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:08.339768 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:08.340896 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0806 04:56:08.341052 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1124,"thread_start_us":959,"threads_started":1}
I20260806 04:56:08.342635 907 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba22c0ce29943759ed6c2cd212127d4 (DEFAULT_TABLE table=test-table [id=f3c27330c5d24332887c76b5cc7e4f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:56:08.344164 907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ba22c0ce29943759ed6c2cd212127d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:08.344561 774 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba22c0ce29943759ed6c2cd212127d4 (DEFAULT_TABLE table=test-table [id=f3c27330c5d24332887c76b5cc7e4f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:56:08.344614 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba22c0ce29943759ed6c2cd212127d4 (DEFAULT_TABLE table=test-table [id=f3c27330c5d24332887c76b5cc7e4f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:56:08.345906 1041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ba22c0ce29943759ed6c2cd212127d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:08.345916 774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ba22c0ce29943759ed6c2cd212127d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:08.356132 1127 tablet_bootstrap.cc:492] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e: Bootstrap starting.
I20260806 04:56:08.358062 1128 tablet_bootstrap.cc:492] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d: Bootstrap starting.
I20260806 04:56:08.360359 1127 tablet_bootstrap.cc:654] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:08.361477 1128 tablet_bootstrap.cc:654] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:08.361573 1127 log.cc:824] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:08.362658 1128 log.cc:824] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:08.364315 1129 tablet_bootstrap.cc:492] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31: Bootstrap starting.
I20260806 04:56:08.365880 1128 tablet_bootstrap.cc:492] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d: No bootstrap required, opened a new log
I20260806 04:56:08.366133 1128 ts_tablet_manager.cc:1397] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 04:56:08.366709 1127 tablet_bootstrap.cc:492] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e: No bootstrap required, opened a new log
I20260806 04:56:08.366983 1127 ts_tablet_manager.cc:1397] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260806 04:56:08.367486 1129 tablet_bootstrap.cc:654] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:08.368603 1129 log.cc:824] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:08.371517 1129 tablet_bootstrap.cc:492] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31: No bootstrap required, opened a new log
I20260806 04:56:08.371768 1129 ts_tablet_manager.cc:1397] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260806 04:56:08.374611 1127 raft_consensus.cc:348] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92befa0bd5a340969cb6914889348f5d" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 38677 } } peers { permanent_uuid: "bb75984a6d5c42e1b8a1bf90d1f7ca31" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36565 } } peers { permanent_uuid: "daeb5a0bcc564851b4b310ed1069158e" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45225 } }
I20260806 04:56:08.374929 1127 raft_consensus.cc:374] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:08.375030 1127 raft_consensus.cc:729] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daeb5a0bcc564851b4b310ed1069158e, State: Initialized, Role: FOLLOWER
I20260806 04:56:08.375571 1127 consensus_queue.cc:260] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92befa0bd5a340969cb6914889348f5d" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 38677 } } peers { permanent_uuid: "bb75984a6d5c42e1b8a1bf90d1f7ca31" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36565 } } peers { permanent_uuid: "daeb5a0bcc564851b4b310ed1069158e" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45225 } }
I20260806 04:56:08.375785 1128 raft_consensus.cc:348] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92befa0bd5a340969cb6914889348f5d" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 38677 } } peers { permanent_uuid: "bb75984a6d5c42e1b8a1bf90d1f7ca31" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36565 } } peers { permanent_uuid: "daeb5a0bcc564851b4b310ed1069158e" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45225 } }
I20260806 04:56:08.376210 1128 raft_consensus.cc:374] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:08.376300 1128 raft_consensus.cc:729] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92befa0bd5a340969cb6914889348f5d, State: Initialized, Role: FOLLOWER
I20260806 04:56:08.376878 1109 heartbeater.cc:503] Master 127.13.148.190:33201 was elected leader, sending a full tablet report...
I20260806 04:56:08.377252 1128 consensus_queue.cc:260] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92befa0bd5a340969cb6914889348f5d" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 38677 } } peers { permanent_uuid: "bb75984a6d5c42e1b8a1bf90d1f7ca31" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36565 } } peers { permanent_uuid: "daeb5a0bcc564851b4b310ed1069158e" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45225 } }
I20260806 04:56:08.378458 975 heartbeater.cc:503] Master 127.13.148.190:33201 was elected leader, sending a full tablet report...
I20260806 04:56:08.378659 1127 ts_tablet_manager.cc:1428] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e: Time spent starting tablet: real 0.012s user 0.004s sys 0.007s
I20260806 04:56:08.380461 1128 ts_tablet_manager.cc:1428] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260806 04:56:08.381206 1129 raft_consensus.cc:348] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92befa0bd5a340969cb6914889348f5d" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 38677 } } peers { permanent_uuid: "bb75984a6d5c42e1b8a1bf90d1f7ca31" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36565 } } peers { permanent_uuid: "daeb5a0bcc564851b4b310ed1069158e" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45225 } }
I20260806 04:56:08.381582 1129 raft_consensus.cc:374] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:08.381697 1129 raft_consensus.cc:729] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb75984a6d5c42e1b8a1bf90d1f7ca31, State: Initialized, Role: FOLLOWER
I20260806 04:56:08.382310 1129 consensus_queue.cc:260] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92befa0bd5a340969cb6914889348f5d" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 38677 } } peers { permanent_uuid: "bb75984a6d5c42e1b8a1bf90d1f7ca31" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36565 } } peers { permanent_uuid: "daeb5a0bcc564851b4b310ed1069158e" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45225 } }
I20260806 04:56:08.383201 841 heartbeater.cc:503] Master 127.13.148.190:33201 was elected leader, sending a full tablet report...
I20260806 04:56:08.383772 1129 ts_tablet_manager.cc:1428] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
W20260806 04:56:08.416925 842 tablet.cc:2369] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:56:08.418246 1134 raft_consensus.cc:482] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:56:08.418505 1134 raft_consensus.cc:504] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92befa0bd5a340969cb6914889348f5d" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 38677 } } peers { permanent_uuid: "bb75984a6d5c42e1b8a1bf90d1f7ca31" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36565 } } peers { permanent_uuid: "daeb5a0bcc564851b4b310ed1069158e" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45225 } }
I20260806 04:56:08.419939 1134 leader_election.cc:288] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb75984a6d5c42e1b8a1bf90d1f7ca31 (127.13.148.129:36565), daeb5a0bcc564851b4b310ed1069158e (127.13.148.131:45225)
I20260806 04:56:08.426214 1124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.420302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:58931 (local address 127.13.148.129:36565)
0806 04:56:08.420481 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:08.420492 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:08.420652 (+ 160us) server_negotiation.cc:407] Connection header received
0806 04:56:08.420895 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:08.420902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:08.421011 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:08.421081 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:08.421831 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.422733 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.424489 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.425356 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.425464 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:08.425517 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:08.425881 (+ 364us) server_negotiation.cc:299] Negotiation successful
0806 04:56:08.426007 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:56:08.426218 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.420106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36565 (local address 127.13.148.130:58931)
0806 04:56:08.420552 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:08.420571 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:08.420690 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:08.421340 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:08.421344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:08.421373 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:08.421688 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:08.421703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.422855 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.422859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:08.424336 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:08.424348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.425509 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.425522 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:08.425711 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:08.425739 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 04:56:08.425828 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 04:56:08.425931 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":59,"thread_start_us":228,"threads_started":1}
I20260806 04:56:08.426286 1126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.420932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:36749 (local address 127.13.148.131:45225)
0806 04:56:08.421097 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:08.421102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:08.421647 (+ 545us) server_negotiation.cc:407] Connection header received
0806 04:56:08.421792 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:08.421799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:08.421867 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:08.421951 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:08.422375 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.423080 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.424752 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.425527 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.425676 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:08.425733 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:08.425973 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 04:56:08.426088 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:56:08.426378 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.420707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:45225 (local address 127.13.148.130:36749)
0806 04:56:08.421534 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:08.421549 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:08.421671 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:08.421976 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:08.421979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:08.422000 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:08.422246 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:08.422253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.423209 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.423213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:08.424582 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:08.424596 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.425690 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.425694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:08.425800 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:08.425850 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 04:56:08.425932 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 04:56:08.426030 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":1,"thread_start_us":523,"threads_started":1}
I20260806 04:56:08.427274 794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba22c0ce29943759ed6c2cd212127d4" candidate_uuid: "92befa0bd5a340969cb6914889348f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb75984a6d5c42e1b8a1bf90d1f7ca31" is_pre_election: true
I20260806 04:56:08.427277 1063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba22c0ce29943759ed6c2cd212127d4" candidate_uuid: "92befa0bd5a340969cb6914889348f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daeb5a0bcc564851b4b310ed1069158e" is_pre_election: true
I20260806 04:56:08.427676 794 raft_consensus.cc:2454] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92befa0bd5a340969cb6914889348f5d in term 0.
I20260806 04:56:08.427719 1063 raft_consensus.cc:2454] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92befa0bd5a340969cb6914889348f5d in term 0.
I20260806 04:56:08.428264 860 leader_election.cc:302] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92befa0bd5a340969cb6914889348f5d, bb75984a6d5c42e1b8a1bf90d1f7ca31; no voters:
I20260806 04:56:08.428604 1134 raft_consensus.cc:2781] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:56:08.428755 1134 raft_consensus.cc:482] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:56:08.428905 1134 raft_consensus.cc:3037] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:08.431308 1134 raft_consensus.cc:504] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92befa0bd5a340969cb6914889348f5d" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 38677 } } peers { permanent_uuid: "bb75984a6d5c42e1b8a1bf90d1f7ca31" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36565 } } peers { permanent_uuid: "daeb5a0bcc564851b4b310ed1069158e" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45225 } }
I20260806 04:56:08.431810 1134 leader_election.cc:288] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [CANDIDATE]: Term 1 election: Requested vote from peers bb75984a6d5c42e1b8a1bf90d1f7ca31 (127.13.148.129:36565), daeb5a0bcc564851b4b310ed1069158e (127.13.148.131:45225)
I20260806 04:56:08.432116 794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba22c0ce29943759ed6c2cd212127d4" candidate_uuid: "92befa0bd5a340969cb6914889348f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb75984a6d5c42e1b8a1bf90d1f7ca31"
I20260806 04:56:08.432256 1063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba22c0ce29943759ed6c2cd212127d4" candidate_uuid: "92befa0bd5a340969cb6914889348f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daeb5a0bcc564851b4b310ed1069158e"
I20260806 04:56:08.432314 794 raft_consensus.cc:3037] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:08.432435 1063 raft_consensus.cc:3037] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:08.434343 794 raft_consensus.cc:2454] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92befa0bd5a340969cb6914889348f5d in term 1.
I20260806 04:56:08.434563 1063 raft_consensus.cc:2454] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92befa0bd5a340969cb6914889348f5d in term 1.
I20260806 04:56:08.434752 860 leader_election.cc:302] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92befa0bd5a340969cb6914889348f5d, bb75984a6d5c42e1b8a1bf90d1f7ca31; no voters:
I20260806 04:56:08.435020 1134 raft_consensus.cc:2781] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:08.435403 1134 raft_consensus.cc:686] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [term 1 LEADER]: Becoming Leader. State: Replica: 92befa0bd5a340969cb6914889348f5d, State: Running, Role: LEADER
I20260806 04:56:08.435814 1134 consensus_queue.cc:237] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92befa0bd5a340969cb6914889348f5d" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 38677 } } peers { permanent_uuid: "bb75984a6d5c42e1b8a1bf90d1f7ca31" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36565 } } peers { permanent_uuid: "daeb5a0bcc564851b4b310ed1069158e" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45225 } }
I20260806 04:56:08.440310 650 catalog_manager.cc:5495] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 92befa0bd5a340969cb6914889348f5d (127.13.148.130). New cstate: current_term: 1 leader_uuid: "92befa0bd5a340969cb6914889348f5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92befa0bd5a340969cb6914889348f5d" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 38677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb75984a6d5c42e1b8a1bf90d1f7ca31" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "daeb5a0bcc564851b4b310ed1069158e" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45225 } health_report { overall_health: UNKNOWN } } }
I20260806 04:56:08.464196 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.459669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.13.148.130:38677)
0806 04:56:08.459828 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:08.459832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:08.459997 (+ 165us) server_negotiation.cc:407] Connection header received
0806 04:56:08.460141 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:08.460144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:08.460206 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:08.460316 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:08.460625 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.461306 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.462498 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.462859 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.462941 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:08.463361 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:08.463789 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:08.463916 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:08.464011 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 04:56:08.464041 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:56:08.471735 794 raft_consensus.cc:1261] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31 [term 1 FOLLOWER]: Refusing update from remote peer 92befa0bd5a340969cb6914889348f5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:56:08.471726 1063 raft_consensus.cc:1261] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e [term 1 FOLLOWER]: Refusing update from remote peer 92befa0bd5a340969cb6914889348f5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:56:08.472628 1134 consensus_queue.cc:1035] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb75984a6d5c42e1b8a1bf90d1f7ca31" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:56:08.472915 1139 consensus_queue.cc:1035] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "daeb5a0bcc564851b4b310ed1069158e" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:56:08.484105 1143 mvcc.cc:205] Tried to move back new op lower bound from 7315423922050957312 to 7315423921916690432. Current Snapshot: MvccSnapshot[applied={T|T < 7315423922050957312}]
I20260806 04:56:08.494890 650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:56:08.495215 650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:56:08.498637 650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbc00cf3f33140dba6821291803420d6: Sending DeleteTablet for 3 replicas of tablet 7ba22c0ce29943759ed6c2cd212127d4
I20260806 04:56:08.499678 907 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ba22c0ce29943759ed6c2cd212127d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:56:08 UTC) from {username='slave'} at 127.0.0.1:50094
I20260806 04:56:08.499800 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ba22c0ce29943759ed6c2cd212127d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:56:08 UTC) from {username='slave'} at 127.0.0.1:51472
I20260806 04:56:08.500030 774 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ba22c0ce29943759ed6c2cd212127d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:56:08 UTC) from {username='slave'} at 127.0.0.1:58420
I20260806 04:56:08.500895 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 711
I20260806 04:56:08.512318 1154 tablet_replica.cc:330] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31: stopping tablet replica
I20260806 04:56:08.512825 1154 raft_consensus.cc:2229] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:56:08.513335 1154 raft_consensus.cc:2258] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:56:08.515594 1154 ts_tablet_manager.cc:1905] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:56:08.517272 1153 tablet_replica.cc:330] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e: stopping tablet replica
I20260806 04:56:08.517663 1153 raft_consensus.cc:2229] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:56:08.518203 1153 raft_consensus.cc:2258] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:56:08.519758 1152 tablet_replica.cc:330] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d: stopping tablet replica
I20260806 04:56:08.520153 1152 raft_consensus.cc:2229] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:56:08.520737 1152 raft_consensus.cc:2258] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:56:08.522703 1152 ts_tablet_manager.cc:1905] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:56:08.522946 1124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.502344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58424 (local address 127.13.148.129:36565)
0806 04:56:08.502554 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:08.502562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:08.513431 (+ 10869us) server_negotiation.cc:407] Connection header received
0806 04:56:08.513910 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:08.513915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:08.514035 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:08.514110 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:08.514817 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.515871 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.517830 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.518489 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.518593 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:08.521395 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:08.521405 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:08.521418 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:08.522113 (+ 695us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:08.522162 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:08.522185 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:08.522270 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:08.522506 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 04:56:08.522604 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:56:08.523864 1154 ts_tablet_manager.cc:1918] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:56:08.523921 835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:08.524045 1154 log.cc:1196] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/7ba22c0ce29943759ed6c2cd212127d4
I20260806 04:56:08.525987 1153 ts_tablet_manager.cc:1905] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:56:08.529800 1152 ts_tablet_manager.cc:1918] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:56:08.529996 1152 log.cc:1196] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/7ba22c0ce29943759ed6c2cd212127d4
I20260806 04:56:08.530557 1152 ts_tablet_manager.cc:1939] T 7ba22c0ce29943759ed6c2cd212127d4 P 92befa0bd5a340969cb6914889348f5d: Deleting consensus metadata
I20260806 04:56:08.532277 636 catalog_manager.cc:4841] TS 92befa0bd5a340969cb6914889348f5d (127.13.148.130:38677): tablet 7ba22c0ce29943759ed6c2cd212127d4 (table test-table [id=f3c27330c5d24332887c76b5cc7e4f18]) successfully deleted
I20260806 04:56:08.534492 1153 ts_tablet_manager.cc:1918] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:56:08.534646 1153 log.cc:1196] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/7ba22c0ce29943759ed6c2cd212127d4
I20260806 04:56:08.535279 1153 ts_tablet_manager.cc:1939] T 7ba22c0ce29943759ed6c2cd212127d4 P daeb5a0bcc564851b4b310ed1069158e: Deleting consensus metadata
I20260806 04:56:08.537590 635 catalog_manager.cc:4841] TS daeb5a0bcc564851b4b310ed1069158e (127.13.148.131:45225): tablet 7ba22c0ce29943759ed6c2cd212127d4 (table test-table [id=f3c27330c5d24332887c76b5cc7e4f18]) successfully deleted
I20260806 04:56:08.621958 1154 ts_tablet_manager.cc:1939] T 7ba22c0ce29943759ed6c2cd212127d4 P bb75984a6d5c42e1b8a1bf90d1f7ca31: Deleting consensus metadata
I20260806 04:56:08.622125 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 711
I20260806 04:56:08.641973 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 844
W20260806 04:56:08.641973 635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.129:36565 (error 108)
I20260806 04:56:08.644516 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.643996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36565 (local address 127.0.0.1:58432)
0806 04:56:08.644170 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:08.644312 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:36565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 04:56:08.644763 635 catalog_manager.cc:4584] TS bb75984a6d5c42e1b8a1bf90d1f7ca31 (127.13.148.129:36565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ba22c0ce29943759ed6c2cd212127d4: Network error: Client connection negotiation failed: client connection to 127.13.148.129:36565: connect: Connection refused (error 111)
I20260806 04:56:08.650307 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.643222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.13.148.130:38677)
0806 04:56:08.643437 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:08.643442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:08.643460 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:56:08.643515 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:08.643519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:08.643593 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:08.643681 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:08.644187 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.645391 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.646684 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.647336 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.647473 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:08.648523 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:08.648536 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:08.648555 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:08.649495 (+ 940us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:08.649569 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:08.649594 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:08.649694 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:08.649829 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 04:56:08.649933 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 04:56:08.651021 969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:08.698518 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.698077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36565 (local address 127.0.0.1:58444)
0806 04:56:08.698282 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:08.698364 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:36565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 04:56:08.699496 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.699180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36565 (local address 127.0.0.1:58456)
0806 04:56:08.699300 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:08.699373 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:36565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 04:56:08.760061 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.759525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36565 (local address 127.0.0.1:58468)
0806 04:56:08.759736 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:08.759827 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:36565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 04:56:08.761405 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.760977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36565 (local address 127.0.0.1:58484)
0806 04:56:08.761173 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:08.761264 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:36565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 04:56:08.761803 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 844
I20260806 04:56:08.781921 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 978
I20260806 04:56:08.788211 1126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.782688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.13.148.131:45225)
0806 04:56:08.782868 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:08.782873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:08.782898 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:56:08.782958 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:08.782962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:08.783040 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:08.783120 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:08.783484 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.784241 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.785453 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:08.785921 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:08.786012 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:08.786910 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:08.786919 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:08.786933 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:08.787511 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:08.787556 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:08.787575 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:08.787656 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:08.787768 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 04:56:08.787847 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 04:56:08.788972 1104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:08.842065 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.841559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36565 (local address 127.0.0.1:58488)
0806 04:56:08.841764 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:08.841864 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:36565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 04:56:08.843159 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:08.842722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36565 (local address 127.0.0.1:58496)
0806 04:56:08.842926 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:08.843017 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:36565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260806 04:56:08.876631 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 978
I20260806 04:56:08.895604 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 620
I20260806 04:56:08.896764 681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:08.999310 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 620
2026-08-06T04:56:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2569 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:09 dist-test-slave-1x4g krb5kdc[1163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:09 dist-test-slave-1x4g krb5kdc[1163](info): set up 2 sockets
Aug 06 04:56:09 dist-test-slave-1x4g krb5kdc[1163](info): commencing operation
krb5kdc: starting...
W20260806 04:56:10.022149 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:10 dist-test-slave-1x4g krb5kdc[1163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:56:10.261946 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:39921
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:39921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:10.565510 1176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:10.565817 1176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:10.565984 1176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:10.572645 1176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:10.572784 1176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:10.572938 1176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:10.573040 1176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:10.573124 1176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:10.582230 1176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:39921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:39921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:10.583423 1176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:10.584611 1176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:10.595610 1181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:10.595656 1184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:10.595692 1182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:10.596626 1176 server_base.cc:1034] running on GCE node
I20260806 04:56:10.597396 1176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:10.598786 1176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:10.600005 1176 hybrid_clock.cc:648] HybridClock initialized: now 1785992170599969 us; error 50 us; skew 500 ppm
I20260806 04:56:10.600325 1176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:10 dist-test-slave-1x4g krb5kdc[1163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992170, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:10.608362 1176 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:56:10.609943 1176 webserver.cc:460] Webserver started at http://127.13.148.190:46263/ using document root <none> and password file <none>
I20260806 04:56:10.610620 1176 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:10.610764 1176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:10.611078 1176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:10.613994 1176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "215b17a2e8014d62b76aadfc8ea23616"
format_stamp: "Formatted at 2026-08-06 04:56:10 on dist-test-slave-1x4g"
server_key: "fe5ebb260899498f12d436d14074067b"
server_key_iv: "cef17a5cd0be950329b42acad386b9e2"
server_key_version: "encryptionkey@0"
I20260806 04:56:10.614782 1176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "215b17a2e8014d62b76aadfc8ea23616"
format_stamp: "Formatted at 2026-08-06 04:56:10 on dist-test-slave-1x4g"
server_key: "fe5ebb260899498f12d436d14074067b"
server_key_iv: "cef17a5cd0be950329b42acad386b9e2"
server_key_version: "encryptionkey@0"
I20260806 04:56:10.619777 1176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260806 04:56:10.622939 1191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:10.624094 1176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 04:56:10.624253 1176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "215b17a2e8014d62b76aadfc8ea23616"
format_stamp: "Formatted at 2026-08-06 04:56:10 on dist-test-slave-1x4g"
server_key: "fe5ebb260899498f12d436d14074067b"
server_key_iv: "cef17a5cd0be950329b42acad386b9e2"
server_key_version: "encryptionkey@0"
I20260806 04:56:10.624428 1176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:10.655189 1176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:10.659706 1176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:10.659979 1176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:10.677255 1176 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:39921
I20260806 04:56:10.677296 1243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:39921 every 8 connection(s)
I20260806 04:56:10.678568 1176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:10.681165 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 1176
I20260806 04:56:10.681339 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:10.681824 13906 external_mini_cluster.cc:1428] Setting key d474910c22b363a538fe1cfb6a5e2c51
I20260806 04:56:10.682374 1244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 04:56:10 dist-test-slave-1x4g krb5kdc[1163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:56:10.695645 1244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616: Bootstrap starting.
I20260806 04:56:10.699589 1244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:10.700762 1244 log.cc:824] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:10.703794 1244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616: No bootstrap required, opened a new log
I20260806 04:56:10.704008 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:10.683543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.13.148.190:39921)
0806 04:56:10.684290 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:10.684309 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:10.684358 (+ 49us) server_negotiation.cc:407] Connection header received
0806 04:56:10.686048 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:10.686071 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:10.686595 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:10.687009 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:10.688275 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:10.690268 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:10.691392 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:10.692281 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:10.692475 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:10.696225 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:10.696234 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:10.696250 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:10.696292 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:10.700149 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:10.700763 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:10.700772 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:10.700807 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:10.700995 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:10.701400 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:10.701405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:10.701407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:10.701894 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:10.702185 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:10.702538 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 04:56:10.702899 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":307,"threads_started":1}
I20260806 04:56:10.711452 1244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215b17a2e8014d62b76aadfc8ea23616" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 39921 } }
I20260806 04:56:10.711738 1244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:10.711805 1244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 215b17a2e8014d62b76aadfc8ea23616, State: Initialized, Role: FOLLOWER
I20260806 04:56:10.712307 1244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215b17a2e8014d62b76aadfc8ea23616" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 39921 } }
I20260806 04:56:10.712499 1244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:10.712587 1244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:10.712698 1244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:10.714536 1244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215b17a2e8014d62b76aadfc8ea23616" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 39921 } }
I20260806 04:56:10.714910 1244 leader_election.cc:302] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 215b17a2e8014d62b76aadfc8ea23616; no voters:
I20260806 04:56:10.715387 1244 leader_election.cc:288] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:10.715523 1249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:10.716109 1249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [term 1 LEADER]: Becoming Leader. State: Replica: 215b17a2e8014d62b76aadfc8ea23616, State: Running, Role: LEADER
I20260806 04:56:10.716620 1249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215b17a2e8014d62b76aadfc8ea23616" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 39921 } }
I20260806 04:56:10.717339 1244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:10.719787 1251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 215b17a2e8014d62b76aadfc8ea23616. Latest consensus state: current_term: 1 leader_uuid: "215b17a2e8014d62b76aadfc8ea23616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215b17a2e8014d62b76aadfc8ea23616" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 39921 } } }
I20260806 04:56:10.719691 1250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "215b17a2e8014d62b76aadfc8ea23616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215b17a2e8014d62b76aadfc8ea23616" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 39921 } } }
I20260806 04:56:10.720278 1250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:10.720955 1251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:10.721858 1258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:10.726693 1258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:10.735049 1258 catalog_manager.cc:1347] Generated new cluster ID: 4f314a3ac9234d4891962e83e6a5d413
I20260806 04:56:10.735152 1258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:10.760479 1258 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:10.762135 1258 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:10.782743 1258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616: Generated new TSK 0
I20260806 04:56:10.783725 1258 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
Entry for principal kudu/127.13.148.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab.
I20260806 04:56:10.845263 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:39921
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:11.158622 1270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:11.158991 1270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:11.159183 1270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:11.166066 1270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:11.166163 1270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:11.166297 1270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:56:11.175244 1270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.129.keytab
--principal=kudu/127.13.148.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:39921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:11.176409 1270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:11.177768 1270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:11.189776 1276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:11.189713 1278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:11.189742 1275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:11.190785 1270 server_base.cc:1034] running on GCE node
I20260806 04:56:11.191427 1270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:11.192236 1270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:11.193441 1270 hybrid_clock.cc:648] HybridClock initialized: now 1785992171193406 us; error 47 us; skew 500 ppm
I20260806 04:56:11.193717 1270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:11 dist-test-slave-1x4g krb5kdc[1163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992171, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:11.196445 1270 init.cc:377] Logged in from keytab as kudu/127.13.148.129@KRBTEST.COM (short username kudu)
I20260806 04:56:11.197813 1270 webserver.cc:460] Webserver started at http://127.13.148.129:46075/ using document root <none> and password file <none>
I20260806 04:56:11.198438 1270 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:11.198534 1270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:11.198784 1270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:11.201413 1270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "c81bdb37e71b44a3933469a5bb177bac"
format_stamp: "Formatted at 2026-08-06 04:56:11 on dist-test-slave-1x4g"
server_key: "10100ed0316a1a79890c90504c5899ce"
server_key_iv: "5bfec19b71562d070e569218460f0761"
server_key_version: "encryptionkey@0"
I20260806 04:56:11.202020 1270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "c81bdb37e71b44a3933469a5bb177bac"
format_stamp: "Formatted at 2026-08-06 04:56:11 on dist-test-slave-1x4g"
server_key: "10100ed0316a1a79890c90504c5899ce"
server_key_iv: "5bfec19b71562d070e569218460f0761"
server_key_version: "encryptionkey@0"
I20260806 04:56:11.206499 1270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:56:11.209559 1285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:11.210772 1270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:56:11.210950 1270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "c81bdb37e71b44a3933469a5bb177bac"
format_stamp: "Formatted at 2026-08-06 04:56:11 on dist-test-slave-1x4g"
server_key: "10100ed0316a1a79890c90504c5899ce"
server_key_iv: "5bfec19b71562d070e569218460f0761"
server_key_version: "encryptionkey@0"
I20260806 04:56:11.211143 1270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:11.228093 1270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:11.231729 1270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:11.231987 1270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:11.232837 1270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:56:11.233979 1270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:56:11.234042 1270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:11.234194 1270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:56:11.234239 1270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:11.264710 1270 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:36347
I20260806 04:56:11.264755 1404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:36347 every 8 connection(s)
I20260806 04:56:11.265918 1270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:56:11.270903 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 1270
I20260806 04:56:11.271063 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:56:11.271625 13906 external_mini_cluster.cc:1428] Setting key 3a3a24fa1b403053a326ba7a6672b3e4
Aug 06 04:56:11 dist-test-slave-1x4g krb5kdc[1163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992171, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.129@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:56:11.284662 1407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:11.268220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:40181 (local address 127.13.148.190:39921)
0806 04:56:11.268717 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:11.268721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:11.269581 (+ 860us) server_negotiation.cc:407] Connection header received
0806 04:56:11.271264 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:11.271269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:11.271348 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:11.271453 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:11.272981 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:11.273769 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:11.275182 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:11.275537 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:11.275652 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:11.279229 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:11.279234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:11.279238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:11.279268 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:11.281668 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:11.282321 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:11.282328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:11.282335 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:11.282463 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:11.282866 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:11.282870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:11.282872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:11.283249 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:11.283397 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:11.283997 (+ 600us) server_negotiation.cc:299] Negotiation successful
0806 04:56:11.284234 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":292,"threads_started":1}
I20260806 04:56:11.285251 1408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:11.268661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:39921 (local address 127.13.148.129:40181)
0806 04:56:11.269400 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:11.269440 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:11.270940 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:11.271720 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:11.271729 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:11.272177 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:11.272822 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:11.272849 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:11.273915 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:11.273921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:11.275046 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:11.275057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:11.275756 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:11.275764 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:11.275979 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:11.276585 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:11.276618 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:11.278994 (+ 2376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:11.281796 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:11.281803 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:11.281817 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:11.282187 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:11.282572 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:11.282576 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:11.282579 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:11.282758 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:11.283378 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:11.283383 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:11.283653 (+ 270us) client_negotiation.cc:769] Sending connection context
0806 04:56:11.283856 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 04:56:11.284193 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":319,"threads_started":1}
I20260806 04:56:11.286643 1405 heartbeater.cc:347] Connected to a master server at 127.13.148.190:39921
I20260806 04:56:11.286926 1405 heartbeater.cc:464] Registering TS with master...
I20260806 04:56:11.287693 1405 heartbeater.cc:511] Master 127.13.148.190:39921 requested a full tablet report, sending...
I20260806 04:56:11.289816 1209 ts_manager.cc:194] Registered new tserver with Master: c81bdb37e71b44a3933469a5bb177bac (127.13.148.129:36347)
I20260806 04:56:11.291163 1209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.129@KRBTEST.COM'} at 127.13.148.129:40181
WARNING: no policy specified for kudu/127.13.148.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
Entry for principal kudu/127.13.148.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab.
I20260806 04:56:11.329910 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:39921
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:11.634016 1411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:11.634294 1411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:11.634444 1411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:11.641005 1411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:11.641115 1411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:11.641307 1411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:56:11.649638 1411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.130.keytab
--principal=kudu/127.13.148.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:39921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:11.650777 1411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:11.651960 1411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:11.664814 1416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:11.664863 1420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:11.664875 1418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:11.665622 1411 server_base.cc:1034] running on GCE node
I20260806 04:56:11.666155 1411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:11.667028 1411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:11.668237 1411 hybrid_clock.cc:648] HybridClock initialized: now 1785992171668188 us; error 60 us; skew 500 ppm
I20260806 04:56:11.668550 1411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:11 dist-test-slave-1x4g krb5kdc[1163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992171, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:11.671566 1411 init.cc:377] Logged in from keytab as kudu/127.13.148.130@KRBTEST.COM (short username kudu)
I20260806 04:56:11.672966 1411 webserver.cc:460] Webserver started at http://127.13.148.130:43689/ using document root <none> and password file <none>
I20260806 04:56:11.673588 1411 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:11.673717 1411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:11.674041 1411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:11.676695 1411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "23a815bf2e484805925d9c0fc099f851"
format_stamp: "Formatted at 2026-08-06 04:56:11 on dist-test-slave-1x4g"
server_key: "fd1e7f4856a37e225747dfe1b556b2c1"
server_key_iv: "26389ff6c5047bb016094a384da2d17e"
server_key_version: "encryptionkey@0"
I20260806 04:56:11.677500 1411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "23a815bf2e484805925d9c0fc099f851"
format_stamp: "Formatted at 2026-08-06 04:56:11 on dist-test-slave-1x4g"
server_key: "fd1e7f4856a37e225747dfe1b556b2c1"
server_key_iv: "26389ff6c5047bb016094a384da2d17e"
server_key_version: "encryptionkey@0"
I20260806 04:56:11.682374 1411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:56:11.685667 1427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:11.686849 1411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:56:11.687019 1411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "23a815bf2e484805925d9c0fc099f851"
format_stamp: "Formatted at 2026-08-06 04:56:11 on dist-test-slave-1x4g"
server_key: "fd1e7f4856a37e225747dfe1b556b2c1"
server_key_iv: "26389ff6c5047bb016094a384da2d17e"
server_key_version: "encryptionkey@0"
I20260806 04:56:11.687219 1411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:11.700719 1411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:11.704289 1411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:11.704566 1411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:11.705454 1411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:56:11.706602 1411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:56:11.706698 1411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:11.706851 1411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:56:11.706918 1411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:11.737974 1411 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:42887
I20260806 04:56:11.738027 1541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:42887 every 8 connection(s)
I20260806 04:56:11.739225 1411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:56:11.743566 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 1411
I20260806 04:56:11.743774 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:56:11.744323 13906 external_mini_cluster.cc:1428] Setting key d73455627c8954087d6df5cb9f7c98eb
Aug 06 04:56:11 dist-test-slave-1x4g krb5kdc[1163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992171, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.130@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:56:11.760353 1407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:11.741549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:53677 (local address 127.13.148.190:39921)
0806 04:56:11.741739 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:11.741744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:11.742745 (+ 1001us) server_negotiation.cc:407] Connection header received
0806 04:56:11.745001 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:11.745006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:11.745089 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:11.745191 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:11.746803 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:11.747461 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:11.749899 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:11.750239 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:11.750319 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:11.754001 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:11.754009 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:11.754015 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:11.754057 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:11.757370 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:11.758156 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:11.758160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:11.758162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:11.758286 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:11.758793 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:11.758796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:11.758798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:11.759044 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:11.759163 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:11.759966 (+ 803us) server_negotiation.cc:299] Negotiation successful
0806 04:56:11.760110 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 04:56:11.761726 1545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:11.741969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:39921 (local address 127.13.148.130:53677)
0806 04:56:11.742594 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:11.742634 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:11.744693 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:11.745425 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:11.745441 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:11.745901 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:11.746638 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:11.746664 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:11.747617 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:11.747624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:11.749774 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:11.749791 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:11.750434 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:11.750443 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:11.750729 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:11.751379 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:11.751407 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:11.753817 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:11.757502 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:11.757508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:11.757525 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:11.758018 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:11.758414 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:11.758421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:11.758426 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:11.758679 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:11.759171 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:11.759181 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:11.759624 (+ 443us) client_negotiation.cc:769] Sending connection context
0806 04:56:11.759883 (+ 259us) client_negotiation.cc:240] Negotiation successful
0806 04:56:11.760398 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":239,"threads_started":1}
I20260806 04:56:11.763072 1542 heartbeater.cc:347] Connected to a master server at 127.13.148.190:39921
I20260806 04:56:11.763356 1542 heartbeater.cc:464] Registering TS with master...
I20260806 04:56:11.764016 1542 heartbeater.cc:511] Master 127.13.148.190:39921 requested a full tablet report, sending...
I20260806 04:56:11.765307 1209 ts_manager.cc:194] Registered new tserver with Master: 23a815bf2e484805925d9c0fc099f851 (127.13.148.130:42887)
I20260806 04:56:11.766018 1209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.130@KRBTEST.COM'} at 127.13.148.130:53677
WARNING: no policy specified for kudu/127.13.148.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
Entry for principal kudu/127.13.148.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab.
I20260806 04:56:11.801890 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:39921
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:12.103705 1548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:12.103976 1548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:12.104125 1548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:12.110354 1548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:12.110482 1548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:12.110652 1548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:56:12.119241 1548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.131.keytab
--principal=kudu/127.13.148.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:39921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:12.120400 1548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:12.121711 1548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:12.133500 1557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:12.133496 1555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:12.133500 1553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:12.134083 1548 server_base.cc:1034] running on GCE node
I20260806 04:56:12.134575 1548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:12.135386 1548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:12.136606 1548 hybrid_clock.cc:648] HybridClock initialized: now 1785992172136524 us; error 67 us; skew 500 ppm
I20260806 04:56:12.136982 1548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:12 dist-test-slave-1x4g krb5kdc[1163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992172, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:12.139899 1548 init.cc:377] Logged in from keytab as kudu/127.13.148.131@KRBTEST.COM (short username kudu)
I20260806 04:56:12.141208 1548 webserver.cc:460] Webserver started at http://127.13.148.131:46531/ using document root <none> and password file <none>
I20260806 04:56:12.141829 1548 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:12.141965 1548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:12.142241 1548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:12.144838 1548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "72e179bdef6e4cd69d61a32b6db92732"
format_stamp: "Formatted at 2026-08-06 04:56:12 on dist-test-slave-1x4g"
server_key: "33be3e7e401bad9b9cf28219d1590750"
server_key_iv: "d3b773d946cdff8b7ce70fb934ca78de"
server_key_version: "encryptionkey@0"
I20260806 04:56:12.145539 1548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "72e179bdef6e4cd69d61a32b6db92732"
format_stamp: "Formatted at 2026-08-06 04:56:12 on dist-test-slave-1x4g"
server_key: "33be3e7e401bad9b9cf28219d1590750"
server_key_iv: "d3b773d946cdff8b7ce70fb934ca78de"
server_key_version: "encryptionkey@0"
I20260806 04:56:12.150033 1548 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:56:12.153071 1564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:12.154132 1548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:56:12.154280 1548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "72e179bdef6e4cd69d61a32b6db92732"
format_stamp: "Formatted at 2026-08-06 04:56:12 on dist-test-slave-1x4g"
server_key: "33be3e7e401bad9b9cf28219d1590750"
server_key_iv: "d3b773d946cdff8b7ce70fb934ca78de"
server_key_version: "encryptionkey@0"
I20260806 04:56:12.154445 1548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:12.189671 1548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:12.193442 1548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:12.193708 1548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:12.194604 1548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:56:12.195780 1548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:56:12.195878 1548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:12.196018 1548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:56:12.196084 1548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:12.226336 1548 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:45347
I20260806 04:56:12.226377 1678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:45347 every 8 connection(s)
I20260806 04:56:12.227543 1548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:56:12.234694 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 1548
I20260806 04:56:12.234853 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:56:12.235261 13906 external_mini_cluster.cc:1428] Setting key 199414546a3187b1b6d8a833fb732d7a
Aug 06 04:56:12 dist-test-slave-1x4g krb5kdc[1163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992172, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.131@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:56:12.245425 1407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:12.229943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:55751 (local address 127.13.148.190:39921)
0806 04:56:12.230103 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:12.230107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:12.231104 (+ 997us) server_negotiation.cc:407] Connection header received
0806 04:56:12.232763 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:12.232768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:12.232874 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:12.232979 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:12.234260 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.235011 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.236510 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.237024 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.237157 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:12.240370 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:12.240375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:12.240378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:12.240406 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:12.242634 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:12.243273 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:12.243279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:12.243286 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:12.243396 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:12.243794 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:12.243798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:12.243801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:12.244059 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:12.244211 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:12.244848 (+ 637us) server_negotiation.cc:299] Negotiation successful
0806 04:56:12.245099 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:56:12.246194 1681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:12.230364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:39921 (local address 127.13.148.131:55751)
0806 04:56:12.230969 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:12.231003 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:12.232521 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:12.233188 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:12.233195 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:12.233627 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:12.234106 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:12.234121 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.235156 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.235164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:12.236371 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:12.236381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.237240 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.237249 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:12.237509 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:12.238149 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:12.238169 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:12.240175 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:12.242764 (+ 2589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:12.242771 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:12.242784 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:12.243154 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:12.243510 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:12.243514 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:12.243518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:12.243679 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:12.244194 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:12.244200 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:12.244503 (+ 303us) client_negotiation.cc:769] Sending connection context
0806 04:56:12.244735 (+ 232us) client_negotiation.cc:240] Negotiation successful
0806 04:56:12.245219 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":250,"threads_started":1}
I20260806 04:56:12.247315 1679 heartbeater.cc:347] Connected to a master server at 127.13.148.190:39921
I20260806 04:56:12.247584 1679 heartbeater.cc:464] Registering TS with master...
I20260806 04:56:12.248226 1679 heartbeater.cc:511] Master 127.13.148.190:39921 requested a full tablet report, sending...
I20260806 04:56:12.249897 1209 ts_manager.cc:194] Registered new tserver with Master: 72e179bdef6e4cd69d61a32b6db92732 (127.13.148.131:45347)
I20260806 04:56:12.250402 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:56:12.250797 1209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.131@KRBTEST.COM'} at 127.13.148.131:55751
I20260806 04:56:12.264740 1407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:12.254406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48824 (local address 127.13.148.190:39921)
0806 04:56:12.254548 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:12.254551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:12.254911 (+ 360us) server_negotiation.cc:407] Connection header received
0806 04:56:12.255018 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:12.255021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:12.255067 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:12.255132 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:12.256295 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.256987 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.258191 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.258600 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.258685 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:12.260682 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:12.260686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:12.260689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:12.260713 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:12.262694 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:12.263255 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:12.263262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:12.263266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:12.263401 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:12.263689 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:12.263691 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:12.263693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:12.263934 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:12.264125 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:12.264441 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 04:56:12.264569 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:56:12.268177 1209 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:56:12.274209 1209 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:56:12.294384 1405 heartbeater.cc:503] Master 127.13.148.190:39921 was elected leader, sending a full tablet report...
I20260806 04:56:12.324456 1695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:12.285272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.13.148.130:42887)
0806 04:56:12.315292 (+ 30020us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:12.315302 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:12.316308 (+ 1006us) server_negotiation.cc:407] Connection header received
0806 04:56:12.316475 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:12.316487 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:12.316858 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:12.317145 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:12.318658 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.319725 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.322013 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.323278 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.323426 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:12.323603 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:12.323969 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 04:56:12.324128 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27858,"thread_start_us":27709,"threads_started":1}
I20260806 04:56:12.324582 1694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:12.284428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:42887 (local address 127.0.0.1:34480)
0806 04:56:12.315729 (+ 31301us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:12.315762 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:12.316013 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:12.317826 (+ 1813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:12.317830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:12.317858 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:12.318361 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:12.318384 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.319845 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.319851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:12.321862 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:12.321875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.323485 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.323491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:12.323681 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:12.323736 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 04:56:12.323873 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 04:56:12.324090 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31133,"mutex_wait_us":25,"thread_start_us":31365,"threads_started":1}
I20260806 04:56:12.324766 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:12.285967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51946 (local address 127.13.148.129:36347)
0806 04:56:12.314990 (+ 29023us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:12.315001 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:12.316440 (+ 1439us) server_negotiation.cc:407] Connection header received
0806 04:56:12.316553 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:12.316561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:12.316845 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:12.317033 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:12.318019 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.319698 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.321911 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.323285 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.323461 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:12.323603 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:12.324284 (+ 681us) server_negotiation.cc:299] Negotiation successful
0806 04:56:12.324465 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28817,"thread_start_us":28621,"threads_started":1}
I20260806 04:56:12.324934 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:12.284364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:45347 (local address 127.0.0.1:33550)
0806 04:56:12.299992 (+ 15628us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:12.300054 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:12.300216 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:12.318364 (+ 18148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:12.318369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:12.318399 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:12.318791 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:12.318805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.320831 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.320836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:12.322458 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:12.322470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.324154 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.324159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:12.324326 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:12.324352 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:56:12.324423 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 04:56:12.324595 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15367,"thread_start_us":21670,"threads_started":1}
I20260806 04:56:12.325280 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:12.284885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36347 (local address 127.0.0.1:51946)
0806 04:56:12.316181 (+ 31296us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:12.316210 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:12.316322 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:12.317318 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:12.317328 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:12.317374 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:12.317787 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:12.317813 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.319852 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.319857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:12.321708 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:12.321722 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.323750 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.323754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:12.323895 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:12.323939 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 04:56:12.324022 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 04:56:12.324137 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31136,"mutex_wait_us":55,"thread_start_us":30497,"threads_started":1}
I20260806 04:56:12.325802 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:12.284505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33550 (local address 127.13.148.131:45347)
0806 04:56:12.317783 (+ 33278us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:12.317805 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:12.317836 (+ 31us) server_negotiation.cc:407] Connection header received
0806 04:56:12.317937 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:12.317944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:12.318149 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:12.318969 (+ 820us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:12.319235 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.320690 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.322640 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.323945 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.325208 (+ 1263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:12.325368 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:12.325448 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 04:56:12.325583 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33086,"thread_start_us":32910,"threads_started":1}
I20260806 04:56:12.328685 1336 tablet_service.cc:1467] Processing CreateTablet for tablet 43ecd0e08d6e45258b612d8a397710fb (DEFAULT_TABLE table=test-table [id=7bba59edf9c747bb990b7fe42a2389bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:56:12.328670 1476 tablet_service.cc:1467] Processing CreateTablet for tablet 43ecd0e08d6e45258b612d8a397710fb (DEFAULT_TABLE table=test-table [id=7bba59edf9c747bb990b7fe42a2389bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:56:12.328984 1612 tablet_service.cc:1467] Processing CreateTablet for tablet 43ecd0e08d6e45258b612d8a397710fb (DEFAULT_TABLE table=test-table [id=7bba59edf9c747bb990b7fe42a2389bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:56:12.329960 1336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43ecd0e08d6e45258b612d8a397710fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:12.330061 1476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43ecd0e08d6e45258b612d8a397710fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:12.330157 1612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43ecd0e08d6e45258b612d8a397710fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:12.340411 1697 tablet_bootstrap.cc:492] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac: Bootstrap starting.
I20260806 04:56:12.342921 1698 tablet_bootstrap.cc:492] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851: Bootstrap starting.
I20260806 04:56:12.343655 1697 tablet_bootstrap.cc:654] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:12.344928 1697 log.cc:824] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:12.346169 1698 tablet_bootstrap.cc:654] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:12.347460 1698 log.cc:824] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:12.347733 1699 tablet_bootstrap.cc:492] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732: Bootstrap starting.
I20260806 04:56:12.348189 1697 tablet_bootstrap.cc:492] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac: No bootstrap required, opened a new log
I20260806 04:56:12.348431 1697 ts_tablet_manager.cc:1397] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260806 04:56:12.350668 1698 tablet_bootstrap.cc:492] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851: No bootstrap required, opened a new log
I20260806 04:56:12.350751 1699 tablet_bootstrap.cc:654] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:12.350915 1698 ts_tablet_manager.cc:1397] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 04:56:12.351689 1699 log.cc:824] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:12.354000 1699 tablet_bootstrap.cc:492] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732: No bootstrap required, opened a new log
I20260806 04:56:12.354233 1699 ts_tablet_manager.cc:1397] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 04:56:12.358235 1697 raft_consensus.cc:348] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e179bdef6e4cd69d61a32b6db92732" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45347 } } peers { permanent_uuid: "23a815bf2e484805925d9c0fc099f851" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42887 } } peers { permanent_uuid: "c81bdb37e71b44a3933469a5bb177bac" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36347 } }
I20260806 04:56:12.358659 1697 raft_consensus.cc:374] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:12.358759 1697 raft_consensus.cc:729] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c81bdb37e71b44a3933469a5bb177bac, State: Initialized, Role: FOLLOWER
I20260806 04:56:12.359383 1697 consensus_queue.cc:260] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e179bdef6e4cd69d61a32b6db92732" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45347 } } peers { permanent_uuid: "23a815bf2e484805925d9c0fc099f851" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42887 } } peers { permanent_uuid: "c81bdb37e71b44a3933469a5bb177bac" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36347 } }
I20260806 04:56:12.361110 1698 raft_consensus.cc:348] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e179bdef6e4cd69d61a32b6db92732" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45347 } } peers { permanent_uuid: "23a815bf2e484805925d9c0fc099f851" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42887 } } peers { permanent_uuid: "c81bdb37e71b44a3933469a5bb177bac" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36347 } }
I20260806 04:56:12.361374 1699 raft_consensus.cc:348] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e179bdef6e4cd69d61a32b6db92732" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45347 } } peers { permanent_uuid: "23a815bf2e484805925d9c0fc099f851" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42887 } } peers { permanent_uuid: "c81bdb37e71b44a3933469a5bb177bac" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36347 } }
I20260806 04:56:12.361588 1698 raft_consensus.cc:374] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:12.361691 1698 raft_consensus.cc:729] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a815bf2e484805925d9c0fc099f851, State: Initialized, Role: FOLLOWER
I20260806 04:56:12.361730 1699 raft_consensus.cc:374] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:12.361817 1699 raft_consensus.cc:729] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e179bdef6e4cd69d61a32b6db92732, State: Initialized, Role: FOLLOWER
I20260806 04:56:12.362360 1699 consensus_queue.cc:260] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e179bdef6e4cd69d61a32b6db92732" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45347 } } peers { permanent_uuid: "23a815bf2e484805925d9c0fc099f851" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42887 } } peers { permanent_uuid: "c81bdb37e71b44a3933469a5bb177bac" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36347 } }
I20260806 04:56:12.362313 1698 consensus_queue.cc:260] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e179bdef6e4cd69d61a32b6db92732" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45347 } } peers { permanent_uuid: "23a815bf2e484805925d9c0fc099f851" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42887 } } peers { permanent_uuid: "c81bdb37e71b44a3933469a5bb177bac" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36347 } }
I20260806 04:56:12.363365 1679 heartbeater.cc:503] Master 127.13.148.190:39921 was elected leader, sending a full tablet report...
I20260806 04:56:12.363816 1699 ts_tablet_manager.cc:1428] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260806 04:56:12.367722 1697 ts_tablet_manager.cc:1428] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac: Time spent starting tablet: real 0.019s user 0.012s sys 0.006s
I20260806 04:56:12.368676 1542 heartbeater.cc:503] Master 127.13.148.190:39921 was elected leader, sending a full tablet report...
I20260806 04:56:12.369256 1698 ts_tablet_manager.cc:1428] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
W20260806 04:56:12.479127 1680 tablet.cc:2369] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:56:12.491016 1544 tablet.cc:2369] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:56:12.518072 1406 tablet.cc:2369] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:56:12.683221 1705 raft_consensus.cc:482] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:56:12.683531 1705 raft_consensus.cc:504] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e179bdef6e4cd69d61a32b6db92732" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45347 } } peers { permanent_uuid: "23a815bf2e484805925d9c0fc099f851" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42887 } } peers { permanent_uuid: "c81bdb37e71b44a3933469a5bb177bac" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36347 } }
I20260806 04:56:12.685251 1705 leader_election.cc:288] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72e179bdef6e4cd69d61a32b6db92732 (127.13.148.131:45347), c81bdb37e71b44a3933469a5bb177bac (127.13.148.129:36347)
I20260806 04:56:12.692472 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:12.686064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:52427 (local address 127.13.148.129:36347)
0806 04:56:12.686232 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:12.686237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:12.686792 (+ 555us) server_negotiation.cc:407] Connection header received
0806 04:56:12.686962 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:12.686967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:12.687043 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:12.687137 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:12.687763 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.688702 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.690416 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.691498 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.691636 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:12.691698 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:12.692135 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 04:56:12.692250 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:56:12.692473 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:12.685703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:53117 (local address 127.13.148.131:45347)
0806 04:56:12.685912 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:12.685920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:12.686518 (+ 598us) server_negotiation.cc:407] Connection header received
0806 04:56:12.686802 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:12.686807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:12.686877 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:12.686959 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:12.687502 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.688430 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.690475 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.691511 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.691635 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:12.691698 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:12.692135 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 04:56:12.692250 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 04:56:12.692684 1706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:12.685443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:45347 (local address 127.13.148.130:53117)
0806 04:56:12.686413 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:12.686433 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:12.686637 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:12.686976 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:12.686980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:12.687004 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:12.687372 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:12.687383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.688600 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.688605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:12.690349 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:12.690363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.691775 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.691779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:12.691967 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:12.692008 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 04:56:12.692094 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 04:56:12.692255 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":128,"thread_start_us":654,"threads_started":1}
I20260806 04:56:12.692723 1707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:12.685651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:36347 (local address 127.13.148.130:52427)
0806 04:56:12.686711 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:12.686726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:12.686821 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:12.687174 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:12.687178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:12.687200 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:12.687595 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:12.687605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.688869 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.688881 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:12.690256 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:12.690269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.691772 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.691778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:12.691963 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:12.692024 (+ 61us) client_negotiation.cc:769] Sending connection context
0806 04:56:12.692094 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 04:56:12.692255 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"mutex_wait_us":59,"thread_start_us":764,"threads_started":1}
I20260806 04:56:12.693686 1632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43ecd0e08d6e45258b612d8a397710fb" candidate_uuid: "23a815bf2e484805925d9c0fc099f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e179bdef6e4cd69d61a32b6db92732" is_pre_election: true
I20260806 04:56:12.693725 1357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43ecd0e08d6e45258b612d8a397710fb" candidate_uuid: "23a815bf2e484805925d9c0fc099f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c81bdb37e71b44a3933469a5bb177bac" is_pre_election: true
I20260806 04:56:12.694211 1632 raft_consensus.cc:2454] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a815bf2e484805925d9c0fc099f851 in term 0.
I20260806 04:56:12.694212 1357 raft_consensus.cc:2454] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a815bf2e484805925d9c0fc099f851 in term 0.
I20260806 04:56:12.694847 1432 leader_election.cc:302] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a815bf2e484805925d9c0fc099f851, c81bdb37e71b44a3933469a5bb177bac; no voters:
I20260806 04:56:12.695218 1705 raft_consensus.cc:2781] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:56:12.695349 1705 raft_consensus.cc:482] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:56:12.695456 1705 raft_consensus.cc:3037] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:12.697686 1705 raft_consensus.cc:504] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e179bdef6e4cd69d61a32b6db92732" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45347 } } peers { permanent_uuid: "23a815bf2e484805925d9c0fc099f851" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42887 } } peers { permanent_uuid: "c81bdb37e71b44a3933469a5bb177bac" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36347 } }
I20260806 04:56:12.698241 1705 leader_election.cc:288] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [CANDIDATE]: Term 1 election: Requested vote from peers 72e179bdef6e4cd69d61a32b6db92732 (127.13.148.131:45347), c81bdb37e71b44a3933469a5bb177bac (127.13.148.129:36347)
I20260806 04:56:12.698684 1357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43ecd0e08d6e45258b612d8a397710fb" candidate_uuid: "23a815bf2e484805925d9c0fc099f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c81bdb37e71b44a3933469a5bb177bac"
I20260806 04:56:12.698709 1632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43ecd0e08d6e45258b612d8a397710fb" candidate_uuid: "23a815bf2e484805925d9c0fc099f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e179bdef6e4cd69d61a32b6db92732"
I20260806 04:56:12.698876 1357 raft_consensus.cc:3037] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:12.698887 1632 raft_consensus.cc:3037] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:12.700912 1632 raft_consensus.cc:2454] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a815bf2e484805925d9c0fc099f851 in term 1.
I20260806 04:56:12.700958 1357 raft_consensus.cc:2454] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a815bf2e484805925d9c0fc099f851 in term 1.
I20260806 04:56:12.701254 1429 leader_election.cc:302] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a815bf2e484805925d9c0fc099f851, 72e179bdef6e4cd69d61a32b6db92732; no voters:
I20260806 04:56:12.701519 1705 raft_consensus.cc:2781] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:12.701926 1705 raft_consensus.cc:686] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [term 1 LEADER]: Becoming Leader. State: Replica: 23a815bf2e484805925d9c0fc099f851, State: Running, Role: LEADER
I20260806 04:56:12.702436 1705 consensus_queue.cc:237] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e179bdef6e4cd69d61a32b6db92732" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45347 } } peers { permanent_uuid: "23a815bf2e484805925d9c0fc099f851" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42887 } } peers { permanent_uuid: "c81bdb37e71b44a3933469a5bb177bac" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36347 } }
I20260806 04:56:12.706864 1208 catalog_manager.cc:5495] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23a815bf2e484805925d9c0fc099f851 (127.13.148.130). New cstate: current_term: 1 leader_uuid: "23a815bf2e484805925d9c0fc099f851" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e179bdef6e4cd69d61a32b6db92732" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a815bf2e484805925d9c0fc099f851" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 42887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c81bdb37e71b44a3933469a5bb177bac" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36347 } health_report { overall_health: UNKNOWN } } }
I20260806 04:56:12.770536 1695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:12.765617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.13.148.130:42887)
0806 04:56:12.765765 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:12.765770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:12.766090 (+ 320us) server_negotiation.cc:407] Connection header received
0806 04:56:12.766239 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:12.766243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:12.766312 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:12.766428 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:12.766794 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.767411 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.768666 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.769114 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.769207 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:12.769635 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:12.770093 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:12.770192 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:12.770301 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 04:56:12.770356 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:56:12.777582 1357 raft_consensus.cc:1261] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac [term 1 FOLLOWER]: Refusing update from remote peer 23a815bf2e484805925d9c0fc099f851: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:56:12.777598 1632 raft_consensus.cc:1261] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732 [term 1 FOLLOWER]: Refusing update from remote peer 23a815bf2e484805925d9c0fc099f851: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:56:12.778535 1708 consensus_queue.cc:1035] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e179bdef6e4cd69d61a32b6db92732" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:56:12.778857 1705 consensus_queue.cc:1035] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c81bdb37e71b44a3933469a5bb177bac" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:56:12.800313 1209 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:56:12.800645 1209 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:56:12.804104 1209 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 215b17a2e8014d62b76aadfc8ea23616: Sending DeleteTablet for 3 replicas of tablet 43ecd0e08d6e45258b612d8a397710fb
I20260806 04:56:12.805065 1612 tablet_service.cc:1514] Processing DeleteTablet for tablet 43ecd0e08d6e45258b612d8a397710fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:56:12 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:33550
I20260806 04:56:12.805094 1476 tablet_service.cc:1514] Processing DeleteTablet for tablet 43ecd0e08d6e45258b612d8a397710fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:56:12 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:34480
I20260806 04:56:12.805088 1336 tablet_service.cc:1514] Processing DeleteTablet for tablet 43ecd0e08d6e45258b612d8a397710fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:56:12 UTC) from {username='kudu', principal='kudu/127.13.148.190@KRBTEST.COM'} at 127.0.0.1:51946
I20260806 04:56:12.809787 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 1270
I20260806 04:56:12.818106 1722 tablet_replica.cc:330] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732: stopping tablet replica
I20260806 04:56:12.818696 1724 tablet_replica.cc:330] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac: stopping tablet replica
I20260806 04:56:12.819078 1724 raft_consensus.cc:2229] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:56:12.819578 1724 raft_consensus.cc:2258] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:56:12.820856 1722 raft_consensus.cc:2229] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:56:12.821254 1722 raft_consensus.cc:2258] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:56:12.823858 1722 ts_tablet_manager.cc:1905] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:56:12.823869 1724 ts_tablet_manager.cc:1905] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:56:12.826571 1723 tablet_replica.cc:330] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851: stopping tablet replica
Aug 06 04:56:12 dist-test-slave-1x4g krb5kdc[1163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.129@KRBTEST.COM
I20260806 04:56:12.830950 1724 ts_tablet_manager.cc:1918] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:56:12.831187 1724 log.cc:1196] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-0/wal/wals/43ecd0e08d6e45258b612d8a397710fb
I20260806 04:56:12.831801 1724 ts_tablet_manager.cc:1939] T 43ecd0e08d6e45258b612d8a397710fb P c81bdb37e71b44a3933469a5bb177bac: Deleting consensus metadata
I20260806 04:56:12.833618 1195 catalog_manager.cc:4841] TS c81bdb37e71b44a3933469a5bb177bac (127.13.148.129:36347): tablet 43ecd0e08d6e45258b612d8a397710fb (table test-table [id=7bba59edf9c747bb990b7fe42a2389bc]) successfully deleted
I20260806 04:56:12.834460 1723 raft_consensus.cc:2229] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:56:12.835172 1723 raft_consensus.cc:2258] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:56:12.837010 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:12.811065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.13.148.129:36347)
0806 04:56:12.811223 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:12.811229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:12.813582 (+ 2353us) server_negotiation.cc:407] Connection header received
0806 04:56:12.813778 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:12.813783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:12.813892 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:12.813995 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:12.819477 (+ 5482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.820728 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.822361 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.822944 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.823062 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:12.828647 (+ 5585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:12.828665 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:12.828687 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:12.828741 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:12.832609 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:12.833966 (+ 1357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:12.833977 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:12.833982 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:12.834168 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:12.834664 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:12.834669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:12.834672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:12.835109 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:12.835399 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:12.836406 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0806 04:56:12.836607 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:56:12.837868 1400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:12.838619 1723 ts_tablet_manager.cc:1905] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:56:12.845479 1722 ts_tablet_manager.cc:1918] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:56:12.845763 1722 log.cc:1196] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/43ecd0e08d6e45258b612d8a397710fb
I20260806 04:56:12.846352 1723 ts_tablet_manager.cc:1918] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:56:12.846383 1722 ts_tablet_manager.cc:1939] T 43ecd0e08d6e45258b612d8a397710fb P 72e179bdef6e4cd69d61a32b6db92732: Deleting consensus metadata
I20260806 04:56:12.846547 1723 log.cc:1196] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/43ecd0e08d6e45258b612d8a397710fb
I20260806 04:56:12.847129 1723 ts_tablet_manager.cc:1939] T 43ecd0e08d6e45258b612d8a397710fb P 23a815bf2e484805925d9c0fc099f851: Deleting consensus metadata
I20260806 04:56:12.848304 1193 catalog_manager.cc:4841] TS 72e179bdef6e4cd69d61a32b6db92732 (127.13.148.131:45347): tablet 43ecd0e08d6e45258b612d8a397710fb (table test-table [id=7bba59edf9c747bb990b7fe42a2389bc]) successfully deleted
I20260806 04:56:12.848846 1194 catalog_manager.cc:4841] TS 23a815bf2e484805925d9c0fc099f851 (127.13.148.130:42887): tablet 43ecd0e08d6e45258b612d8a397710fb (table test-table [id=7bba59edf9c747bb990b7fe42a2389bc]) successfully deleted
I20260806 04:56:12.924832 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 1270
I20260806 04:56:12.949075 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 1411
Aug 06 04:56:12 dist-test-slave-1x4g krb5kdc[1163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.130@KRBTEST.COM
I20260806 04:56:12.966346 1695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:12.949983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34496 (local address 127.13.148.130:42887)
0806 04:56:12.950150 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:12.950155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:12.950185 (+ 30us) server_negotiation.cc:407] Connection header received
0806 04:56:12.950313 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:12.950318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:12.950377 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:12.950465 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:12.952181 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.953123 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.954802 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:12.955188 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:12.955273 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:12.960593 (+ 5320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:12.960604 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:12.960622 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:12.960665 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:12.963635 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:12.964235 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:12.964243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:12.964245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:12.964360 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:12.964705 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:12.964709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:12.964712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:12.965123 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:12.965499 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:12.965743 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 04:56:12.965966 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:56:12.967106 1537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:13.071060 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 1411
I20260806 04:56:13.090435 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 1548
Aug 06 04:56:13 dist-test-slave-1x4g krb5kdc[1163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.131@KRBTEST.COM
I20260806 04:56:13.104641 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:13.091111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.13.148.131:45347)
0806 04:56:13.091274 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:13.091281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:13.091302 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:56:13.091521 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:13.091525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:13.091584 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:13.091696 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:13.092981 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:13.093678 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:13.094883 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:13.095431 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:13.095565 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:13.099942 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:13.099953 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:13.099967 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:13.099997 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:13.102438 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:13.103024 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:13.103030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:13.103032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:13.103151 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:13.103446 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:13.103448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:13.103450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:13.103758 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:13.103956 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:13.104233 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 04:56:13.104376 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:56:13.105300 1674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:13.185407 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 1548
I20260806 04:56:13.202997 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 1176
I20260806 04:56:13.204084 1239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:13.306929 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 1176
2026-08-06T04:56:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4308 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T04:56:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:13Z Disabled control of system clock
I20260806 04:56:13.386301 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:35709
--webserver_interface=127.13.148.190
--webserver_port=0
--builtin_ntp_servers=127.13.148.148:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:35709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:56:13.692041 1733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:13.692322 1733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:13.692399 1733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:13.699069 1733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:13.699188 1733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:13.699252 1733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:13.699362 1733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:13.699443 1733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:13.707691 1733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:35709
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:35709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:13.708966 1733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:13.710654 1733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:13.721570 1741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:13.721609 1738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:13.721629 1739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:13.722415 1733 server_base.cc:1034] running on GCE node
I20260806 04:56:13.723095 1733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:13.724318 1733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:13.725528 1733 hybrid_clock.cc:648] HybridClock initialized: now 1785992173725496 us; error 41 us; skew 500 ppm
I20260806 04:56:13.725829 1733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:56:13.727912 1733 webserver.cc:460] Webserver started at http://127.13.148.190:41291/ using document root <none> and password file <none>
I20260806 04:56:13.728538 1733 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:13.728681 1733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:13.729051 1733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:13.731597 1733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "b7284429e65e4d7b9655fa3a08494892"
format_stamp: "Formatted at 2026-08-06 04:56:13 on dist-test-slave-1x4g"
server_key: "8ba1a560aacef00fff2ba5b1d9a53ebd"
server_key_iv: "469b568e37a4ed96b95ed158c19548cc"
server_key_version: "encryptionkey@0"
I20260806 04:56:13.732286 1733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "b7284429e65e4d7b9655fa3a08494892"
format_stamp: "Formatted at 2026-08-06 04:56:13 on dist-test-slave-1x4g"
server_key: "8ba1a560aacef00fff2ba5b1d9a53ebd"
server_key_iv: "469b568e37a4ed96b95ed158c19548cc"
server_key_version: "encryptionkey@0"
I20260806 04:56:13.736855 1733 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 04:56:13.740031 1747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:13.741295 1733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:56:13.741465 1733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "b7284429e65e4d7b9655fa3a08494892"
format_stamp: "Formatted at 2026-08-06 04:56:13 on dist-test-slave-1x4g"
server_key: "8ba1a560aacef00fff2ba5b1d9a53ebd"
server_key_iv: "469b568e37a4ed96b95ed158c19548cc"
server_key_version: "encryptionkey@0"
I20260806 04:56:13.741650 1733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:13.756399 1733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:13.757251 1733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:13.757552 1733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:13.773429 1733 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:35709
I20260806 04:56:13.773438 1801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:35709 every 8 connection(s)
I20260806 04:56:13.774616 1733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:13.776289 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 1733
I20260806 04:56:13.776472 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:13.776995 13906 external_mini_cluster.cc:1428] Setting key a18b8f4a80e4da25d5018f9bf38f1497
I20260806 04:56:13.778301 1802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:13.790472 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:13.778796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.13.148.190:35709)
0806 04:56:13.779801 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:13.779816 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:13.779862 (+ 46us) server_negotiation.cc:407] Connection header received
0806 04:56:13.781510 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:13.781533 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:13.782059 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:13.782435 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:13.782875 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:13.784622 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:13.785869 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:13.786731 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:13.786919 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:13.787860 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:13.787869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:13.787893 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:13.788510 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:13.788629 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:13.788660 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:13.788769 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:13.789044 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 04:56:13.789367 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":324,"threads_started":1}
I20260806 04:56:13.791531 1802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892: Bootstrap starting.
I20260806 04:56:13.794310 1802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:13.795182 1802 log.cc:824] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:13.797796 1802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892: No bootstrap required, opened a new log
I20260806 04:56:13.804239 1802 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7284429e65e4d7b9655fa3a08494892" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 35709 } }
I20260806 04:56:13.804566 1802 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:13.804659 1802 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7284429e65e4d7b9655fa3a08494892, State: Initialized, Role: FOLLOWER
I20260806 04:56:13.805189 1802 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7284429e65e4d7b9655fa3a08494892" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 35709 } }
I20260806 04:56:13.805420 1802 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:13.805548 1802 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:13.805721 1802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:13.807575 1802 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7284429e65e4d7b9655fa3a08494892" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 35709 } }
I20260806 04:56:13.808001 1802 leader_election.cc:302] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7284429e65e4d7b9655fa3a08494892; no voters:
I20260806 04:56:13.808511 1802 leader_election.cc:288] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:13.808689 1807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:13.809834 1807 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [term 1 LEADER]: Becoming Leader. State: Replica: b7284429e65e4d7b9655fa3a08494892, State: Running, Role: LEADER
I20260806 04:56:13.810297 1807 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7284429e65e4d7b9655fa3a08494892" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 35709 } }
I20260806 04:56:13.810614 1802 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:13.813000 1809 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7284429e65e4d7b9655fa3a08494892. Latest consensus state: current_term: 1 leader_uuid: "b7284429e65e4d7b9655fa3a08494892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7284429e65e4d7b9655fa3a08494892" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 35709 } } }
I20260806 04:56:13.813476 1809 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:13.813323 1808 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7284429e65e4d7b9655fa3a08494892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7284429e65e4d7b9655fa3a08494892" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 35709 } } }
I20260806 04:56:13.813630 1808 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:13.814360 1817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:13.819615 1817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:13.827311 1817 catalog_manager.cc:1347] Generated new cluster ID: 6c04b9b8946b4464861c3d588ddf6312
I20260806 04:56:13.827402 1817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:13.861761 1817 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:13.862829 1817 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:13.870971 1817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892: Generated new TSK 0
I20260806 04:56:13.871661 1817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:13.886775 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.129:0
--local_ip_for_outbound_sockets=127.13.148.129
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:35709
--builtin_ntp_servers=127.13.148.148:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:56:14.198266 1827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:14.198544 1827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:14.198709 1827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:14.205145 1827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:14.205261 1827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:14.205430 1827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.129
I20260806 04:56:14.216516 1827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.129
--webserver_port=0
--tserver_master_addrs=127.13.148.190:35709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.129
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:14.217743 1827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:14.219386 1827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:14.231312 1832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:14.231271 1835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:14.231362 1833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:14.232122 1827 server_base.cc:1034] running on GCE node
I20260806 04:56:14.232638 1827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:14.233564 1827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:14.234771 1827 hybrid_clock.cc:648] HybridClock initialized: now 1785992174234720 us; error 39 us; skew 500 ppm
I20260806 04:56:14.235054 1827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:56:14.237159 1827 webserver.cc:460] Webserver started at http://127.13.148.129:46147/ using document root <none> and password file <none>
I20260806 04:56:14.237834 1827 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:14.237968 1827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:14.238256 1827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:14.241084 1827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-0/data/instance:
uuid: "1cf1378f5e8346fdb6f6873acc9b36fd"
format_stamp: "Formatted at 2026-08-06 04:56:14 on dist-test-slave-1x4g"
server_key: "ac31ca06136e84a0713d18cc69ec3e2b"
server_key_iv: "71c402f69744a590bfa549ab992e0d03"
server_key_version: "encryptionkey@0"
I20260806 04:56:14.241791 1827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance:
uuid: "1cf1378f5e8346fdb6f6873acc9b36fd"
format_stamp: "Formatted at 2026-08-06 04:56:14 on dist-test-slave-1x4g"
server_key: "ac31ca06136e84a0713d18cc69ec3e2b"
server_key_iv: "71c402f69744a590bfa549ab992e0d03"
server_key_version: "encryptionkey@0"
I20260806 04:56:14.246444 1827 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:56:14.249732 1841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:14.251034 1827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:56:14.251202 1827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-0/wal
uuid: "1cf1378f5e8346fdb6f6873acc9b36fd"
format_stamp: "Formatted at 2026-08-06 04:56:14 on dist-test-slave-1x4g"
server_key: "ac31ca06136e84a0713d18cc69ec3e2b"
server_key_iv: "71c402f69744a590bfa549ab992e0d03"
server_key_version: "encryptionkey@0"
I20260806 04:56:14.251385 1827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:14.277621 1827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:14.278442 1827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:14.278748 1827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:14.279706 1827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:56:14.281184 1827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:56:14.281301 1827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:14.281432 1827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:56:14.281563 1827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:14.311260 1827 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.129:42807
I20260806 04:56:14.311334 1954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.129:42807 every 8 connection(s)
I20260806 04:56:14.312464 1827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-0/data/info.pb
I20260806 04:56:14.321877 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 1827
I20260806 04:56:14.322065 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-0/wal/instance
I20260806 04:56:14.322603 13906 external_mini_cluster.cc:1428] Setting key 861be02c3944ae8a5b1732e643c61401
I20260806 04:56:14.324144 1957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:14.314851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:51851 (local address 127.13.148.190:35709)
0806 04:56:14.315254 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:14.315260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:14.316359 (+ 1099us) server_negotiation.cc:407] Connection header received
0806 04:56:14.318171 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:14.318175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:14.318239 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:14.318322 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:14.319310 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:14.319983 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:14.321345 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:14.321774 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:14.321868 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:14.323342 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:14.323350 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:14.323356 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:14.323379 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:14.323431 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:14.323445 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:14.323516 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:14.323760 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 04:56:14.323842 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":187,"threads_started":1}
I20260806 04:56:14.325178 1958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:14.315432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:35709 (local address 127.13.148.129:51851)
0806 04:56:14.316211 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:14.316245 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:14.317942 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:14.318521 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:14.318528 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:14.318626 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:14.319166 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:14.319182 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:14.320109 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:14.320112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:14.321224 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:14.321235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:14.322020 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:14.322030 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:14.322303 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:14.323013 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:56:14.323016 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:14.323063 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:56:14.323148 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:14.323539 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:14.323549 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:14.323555 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 04:56:14.323720 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 04:56:14.323963 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":236,"threads_started":1}
I20260806 04:56:14.325606 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.130:0
--local_ip_for_outbound_sockets=127.13.148.130
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:35709
--builtin_ntp_servers=127.13.148.148:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:56:14.326642 1955 heartbeater.cc:347] Connected to a master server at 127.13.148.190:35709
I20260806 04:56:14.326928 1955 heartbeater.cc:464] Registering TS with master...
I20260806 04:56:14.327610 1955 heartbeater.cc:511] Master 127.13.148.190:35709 requested a full tablet report, sending...
I20260806 04:56:14.329780 1765 ts_manager.cc:194] Registered new tserver with Master: 1cf1378f5e8346fdb6f6873acc9b36fd (127.13.148.129:42807)
I20260806 04:56:14.331813 1765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.129:51851
W20260806 04:56:14.631547 1959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:14.631783 1959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:14.631939 1959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:14.638763 1959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:14.638885 1959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:14.639051 1959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.130
I20260806 04:56:14.646912 1959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.130
--webserver_port=0
--tserver_master_addrs=127.13.148.190:35709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.130
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:14.648063 1959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:14.649936 1959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:14.661919 1964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:14.661880 1965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:14.662130 1967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:14.662740 1959 server_base.cc:1034] running on GCE node
I20260806 04:56:14.663329 1959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:14.664193 1959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:14.665416 1959 hybrid_clock.cc:648] HybridClock initialized: now 1785992174665370 us; error 51 us; skew 500 ppm
I20260806 04:56:14.665704 1959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:56:14.667816 1959 webserver.cc:460] Webserver started at http://127.13.148.130:35127/ using document root <none> and password file <none>
I20260806 04:56:14.668556 1959 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:14.668709 1959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:14.669059 1959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:14.671675 1959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/data/instance:
uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c"
format_stamp: "Formatted at 2026-08-06 04:56:14 on dist-test-slave-1x4g"
server_key: "d5e270510db3965c6a444e6de037663a"
server_key_iv: "d4ded7143628c3d0019f213587b59fea"
server_key_version: "encryptionkey@0"
I20260806 04:56:14.672389 1959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance:
uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c"
format_stamp: "Formatted at 2026-08-06 04:56:14 on dist-test-slave-1x4g"
server_key: "d5e270510db3965c6a444e6de037663a"
server_key_iv: "d4ded7143628c3d0019f213587b59fea"
server_key_version: "encryptionkey@0"
I20260806 04:56:14.676617 1959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 04:56:14.679934 1973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:14.681094 1959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:56:14.681244 1959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/wal
uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c"
format_stamp: "Formatted at 2026-08-06 04:56:14 on dist-test-slave-1x4g"
server_key: "d5e270510db3965c6a444e6de037663a"
server_key_iv: "d4ded7143628c3d0019f213587b59fea"
server_key_version: "encryptionkey@0"
I20260806 04:56:14.681427 1959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:14.691689 1959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:14.692390 1959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:14.692675 1959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:14.693522 1959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:56:14.694690 1959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:56:14.694808 1959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:14.694917 1959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:56:14.694993 1959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:14.725409 1959 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.130:44313
I20260806 04:56:14.725482 2087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.130:44313 every 8 connection(s)
I20260806 04:56:14.726574 1959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/data/info.pb
I20260806 04:56:14.731818 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 1959
I20260806 04:56:14.732002 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/wal/instance
I20260806 04:56:14.732565 13906 external_mini_cluster.cc:1428] Setting key ffc85a7b2799bc76406e6447ca1d4c10
I20260806 04:56:14.735533 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.131:0
--local_ip_for_outbound_sockets=127.13.148.131
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:35709
--builtin_ntp_servers=127.13.148.148:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:56:14.739542 1957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:14.728916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:45387 (local address 127.13.148.190:35709)
0806 04:56:14.729131 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:14.729144 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:14.730420 (+ 1276us) server_negotiation.cc:407] Connection header received
0806 04:56:14.732303 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:14.732308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:14.732375 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:14.732479 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:14.733758 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:14.734702 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:14.736869 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:14.737253 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:14.737373 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:14.738835 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:14.738839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:14.738842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:14.738859 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:14.738889 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:14.738901 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:14.738972 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:14.739239 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 04:56:14.739316 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:56:14.740803 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:14.729405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:35709 (local address 127.13.148.130:45387)
0806 04:56:14.730272 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:14.730308 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:14.732014 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:14.732727 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:14.732736 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:14.732854 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:14.733586 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:14.733610 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:14.734844 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:14.734848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:14.736671 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:14.736689 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:14.737472 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:14.737482 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:14.737784 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:14.738475 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:56:14.738481 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:14.738528 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:56:14.738639 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:14.738997 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:14.739032 (+ 35us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:14.739042 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 04:56:14.739207 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 04:56:14.739581 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":285,"threads_started":1}
I20260806 04:56:14.742048 2088 heartbeater.cc:347] Connected to a master server at 127.13.148.190:35709
I20260806 04:56:14.742322 2088 heartbeater.cc:464] Registering TS with master...
I20260806 04:56:14.742951 2088 heartbeater.cc:511] Master 127.13.148.190:35709 requested a full tablet report, sending...
I20260806 04:56:14.744174 1765 ts_manager.cc:194] Registered new tserver with Master: df3d5e14c06e46e4bd9a7c2da38ecb0c (127.13.148.130:44313)
I20260806 04:56:14.745005 1765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.130:45387
W20260806 04:56:15.051280 2091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:15.051501 2091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:15.051570 2091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:15.057829 2091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:15.057909 2091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:15.058085 2091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.131
I20260806 04:56:15.065930 2091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.131
--webserver_port=0
--tserver_master_addrs=127.13.148.190:35709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.131
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:15.066941 2091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:15.068472 2091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:15.080619 2099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:15.080559 2097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:15.080807 2096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:15.081288 2091 server_base.cc:1034] running on GCE node
I20260806 04:56:15.081736 2091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:15.082510 2091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:15.083689 2091 hybrid_clock.cc:648] HybridClock initialized: now 1785992175083649 us; error 37 us; skew 500 ppm
I20260806 04:56:15.083932 2091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:56:15.085915 2091 webserver.cc:460] Webserver started at http://127.13.148.131:37721/ using document root <none> and password file <none>
I20260806 04:56:15.086503 2091 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:15.086606 2091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:15.086884 2091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:15.089390 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/data/instance:
uuid: "ea0f102d3dac4e259cfa59903189000c"
format_stamp: "Formatted at 2026-08-06 04:56:15 on dist-test-slave-1x4g"
server_key: "cf672051109ba89f369a0b518578a045"
server_key_iv: "43fcc91287a60758540b5a40bcef570b"
server_key_version: "encryptionkey@0"
I20260806 04:56:15.090139 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance:
uuid: "ea0f102d3dac4e259cfa59903189000c"
format_stamp: "Formatted at 2026-08-06 04:56:15 on dist-test-slave-1x4g"
server_key: "cf672051109ba89f369a0b518578a045"
server_key_iv: "43fcc91287a60758540b5a40bcef570b"
server_key_version: "encryptionkey@0"
I20260806 04:56:15.094589 2091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:56:15.097857 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:15.098995 2091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:56:15.099165 2091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/wal
uuid: "ea0f102d3dac4e259cfa59903189000c"
format_stamp: "Formatted at 2026-08-06 04:56:15 on dist-test-slave-1x4g"
server_key: "cf672051109ba89f369a0b518578a045"
server_key_iv: "43fcc91287a60758540b5a40bcef570b"
server_key_version: "encryptionkey@0"
I20260806 04:56:15.099370 2091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:15.128604 2091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:15.129436 2091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:15.129756 2091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:15.130637 2091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:56:15.131928 2091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:56:15.132050 2091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:15.132162 2091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:56:15.132239 2091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:15.163053 2091 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.131:34811
I20260806 04:56:15.163126 2219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.131:34811 every 8 connection(s)
I20260806 04:56:15.164234 2091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/data/info.pb
I20260806 04:56:15.173755 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 2091
I20260806 04:56:15.173924 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/wal/instance
I20260806 04:56:15.174508 13906 external_mini_cluster.cc:1428] Setting key e54d0a7b3ab182b51cb0217baf528a6f
I20260806 04:56:15.175683 1957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.166542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.131:46445 (local address 127.13.148.190:35709)
0806 04:56:15.166748 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:15.166754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:15.167973 (+ 1219us) server_negotiation.cc:407] Connection header received
0806 04:56:15.169878 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:15.169883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:15.169949 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:15.170044 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:15.171047 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.171715 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.173072 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.173425 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.173510 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.174900 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:15.174904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:15.174907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:15.174939 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:15.174981 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:15.174997 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:15.175095 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:15.175288 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 04:56:15.175384 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 04:56:15.176743 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.167053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:35709 (local address 127.13.148.131:46445)
0806 04:56:15.167832 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:15.167871 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:15.169622 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:15.170222 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:15.170229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:15.170320 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:15.170874 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:15.170889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.171833 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.171836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:15.172966 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:15.172976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.173624 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.173630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:15.173890 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.174587 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:56:15.174590 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:15.174638 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:56:15.174723 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:15.175103 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:15.175111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:15.175116 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 04:56:15.175248 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 04:56:15.175581 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":229,"threads_started":1}
I20260806 04:56:15.178083 2220 heartbeater.cc:347] Connected to a master server at 127.13.148.190:35709
I20260806 04:56:15.178331 2220 heartbeater.cc:464] Registering TS with master...
I20260806 04:56:15.179021 2220 heartbeater.cc:511] Master 127.13.148.190:35709 requested a full tablet report, sending...
I20260806 04:56:15.180296 1765 ts_manager.cc:194] Registered new tserver with Master: ea0f102d3dac4e259cfa59903189000c (127.13.148.131:34811)
I20260806 04:56:15.181025 1765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.131:46445
I20260806 04:56:15.189339 13906 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:56:15.197858 1957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.193116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40556 (local address 127.13.148.190:35709)
0806 04:56:15.193302 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:15.193306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:15.193688 (+ 382us) server_negotiation.cc:407] Connection header received
0806 04:56:15.193819 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:15.193822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:15.193882 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:15.193957 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:15.194327 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.194889 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.195943 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.196270 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.196348 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.197311 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:15.197316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:15.197319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:15.197334 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:15.197363 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:15.197374 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:15.197442 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:15.197560 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 04:56:15.197627 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:56:15.201530 1765 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:56:15.207747 1765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:56:15.230631 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.219058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:44313 (local address 127.0.0.1:44602)
0806 04:56:15.220431 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:15.220468 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:15.220625 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:15.221609 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:15.221614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:15.221634 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:15.221851 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:15.221858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.223935 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.223943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:15.226671 (+ 2728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:15.226691 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.229416 (+ 2725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.229426 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:15.229724 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.229795 (+ 71us) client_negotiation.cc:769] Sending connection context
0806 04:56:15.229944 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 04:56:15.230164 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"thread_start_us":1004,"threads_started":1}
I20260806 04:56:15.231606 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.219411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.13.148.130:44313)
0806 04:56:15.219962 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:15.219968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:15.220566 (+ 598us) server_negotiation.cc:407] Connection header received
0806 04:56:15.220884 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:15.220892 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:15.221148 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:15.221340 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:15.221991 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.223741 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.226884 (+ 3143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.228701 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.228961 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.229242 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:15.230911 (+ 1669us) server_negotiation.cc:299] Negotiation successful
0806 04:56:15.231127 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":279,"threads_started":1}
I20260806 04:56:15.238023 2021 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd557641624434192105cb3c024dfb0 (DEFAULT_TABLE table=test-table [id=b3ebeb0b22e443eaa8459ecf2df6720a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:56:15.239624 2021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd557641624434192105cb3c024dfb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:15.239974 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.220845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:42807 (local address 127.0.0.1:59910)
0806 04:56:15.221345 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:15.221360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:15.221457 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:15.233898 (+ 12441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:15.233901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:15.233934 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:15.234245 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:15.234258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.235726 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.235733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:15.237667 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:15.237678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.239214 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.239220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:15.239395 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.239431 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 04:56:15.239527 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 04:56:15.239671 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":22,"thread_start_us":280,"threads_started":1}
I20260806 04:56:15.240633 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.221219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59910 (local address 127.13.148.129:42807)
0806 04:56:15.233298 (+ 12079us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:15.233321 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:15.233358 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:56:15.233466 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:15.233473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:15.233699 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:15.234453 (+ 754us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:15.234659 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.235605 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.237874 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.239008 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.239183 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.239403 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:15.239590 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 04:56:15.239758 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11892,"thread_start_us":11771,"threads_started":1}
I20260806 04:56:15.241070 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.222090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:34811 (local address 127.0.0.1:60986)
0806 04:56:15.232199 (+ 10109us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:15.232221 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:15.232328 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:15.234533 (+ 2205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:15.234537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:15.234558 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:15.234748 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:15.234761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.236393 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.236398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:15.238476 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:15.238491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.240435 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.240441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:15.240615 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.240647 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 04:56:15.240731 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 04:56:15.240867 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9967,"mutex_wait_us":21,"thread_start_us":11280,"threads_started":1}
I20260806 04:56:15.243366 1889 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd557641624434192105cb3c024dfb0 (DEFAULT_TABLE table=test-table [id=b3ebeb0b22e443eaa8459ecf2df6720a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:56:15.244132 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.221827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.13.148.131:34811)
0806 04:56:15.233999 (+ 12172us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:15.234004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:15.234061 (+ 57us) server_negotiation.cc:407] Connection header received
0806 04:56:15.234146 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:15.234153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:15.234358 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:15.234528 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:15.234911 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.236239 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.238651 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.240004 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.240166 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.240351 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:15.241358 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0806 04:56:15.241459 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12015,"thread_start_us":11876,"threads_started":1}
I20260806 04:56:15.244570 1889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd557641624434192105cb3c024dfb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:15.247682 2154 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd557641624434192105cb3c024dfb0 (DEFAULT_TABLE table=test-table [id=b3ebeb0b22e443eaa8459ecf2df6720a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:56:15.249024 2154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd557641624434192105cb3c024dfb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:15.250845 2240 tablet_bootstrap.cc:492] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c: Bootstrap starting.
I20260806 04:56:15.253502 2240 tablet_bootstrap.cc:654] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:15.254580 2240 log.cc:824] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:15.258083 2240 tablet_bootstrap.cc:492] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c: No bootstrap required, opened a new log
I20260806 04:56:15.258360 2240 ts_tablet_manager.cc:1397] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 04:56:15.258904 2242 tablet_bootstrap.cc:492] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd: Bootstrap starting.
I20260806 04:56:15.261598 2243 tablet_bootstrap.cc:492] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c: Bootstrap starting.
I20260806 04:56:15.262497 2242 tablet_bootstrap.cc:654] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:15.263695 2242 log.cc:824] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:15.264293 2243 tablet_bootstrap.cc:654] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:15.265285 2243 log.cc:824] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:15.266345 2242 tablet_bootstrap.cc:492] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd: No bootstrap required, opened a new log
I20260806 04:56:15.266597 2242 ts_tablet_manager.cc:1397] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 04:56:15.267416 2243 tablet_bootstrap.cc:492] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c: No bootstrap required, opened a new log
I20260806 04:56:15.267647 2243 ts_tablet_manager.cc:1397] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:56:15.268149 2240 raft_consensus.cc:348] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44313 } } peers { permanent_uuid: "1cf1378f5e8346fdb6f6873acc9b36fd" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42807 } } peers { permanent_uuid: "ea0f102d3dac4e259cfa59903189000c" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 34811 } }
I20260806 04:56:15.268575 2240 raft_consensus.cc:374] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:15.268702 2240 raft_consensus.cc:729] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3d5e14c06e46e4bd9a7c2da38ecb0c, State: Initialized, Role: FOLLOWER
I20260806 04:56:15.269451 2240 consensus_queue.cc:260] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44313 } } peers { permanent_uuid: "1cf1378f5e8346fdb6f6873acc9b36fd" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42807 } } peers { permanent_uuid: "ea0f102d3dac4e259cfa59903189000c" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 34811 } }
I20260806 04:56:15.270565 2088 heartbeater.cc:503] Master 127.13.148.190:35709 was elected leader, sending a full tablet report...
I20260806 04:56:15.272315 2240 ts_tablet_manager.cc:1428] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c: Time spent starting tablet: real 0.014s user 0.004s sys 0.007s
I20260806 04:56:15.275408 2242 raft_consensus.cc:348] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44313 } } peers { permanent_uuid: "1cf1378f5e8346fdb6f6873acc9b36fd" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42807 } } peers { permanent_uuid: "ea0f102d3dac4e259cfa59903189000c" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 34811 } }
I20260806 04:56:15.275808 2242 raft_consensus.cc:374] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:15.275974 2242 raft_consensus.cc:729] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf1378f5e8346fdb6f6873acc9b36fd, State: Initialized, Role: FOLLOWER
I20260806 04:56:15.276135 2243 raft_consensus.cc:348] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44313 } } peers { permanent_uuid: "1cf1378f5e8346fdb6f6873acc9b36fd" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42807 } } peers { permanent_uuid: "ea0f102d3dac4e259cfa59903189000c" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 34811 } }
I20260806 04:56:15.276510 2243 raft_consensus.cc:374] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:15.276475 2242 consensus_queue.cc:260] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44313 } } peers { permanent_uuid: "1cf1378f5e8346fdb6f6873acc9b36fd" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42807 } } peers { permanent_uuid: "ea0f102d3dac4e259cfa59903189000c" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 34811 } }
I20260806 04:56:15.276640 2243 raft_consensus.cc:729] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0f102d3dac4e259cfa59903189000c, State: Initialized, Role: FOLLOWER
I20260806 04:56:15.277280 2243 consensus_queue.cc:260] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44313 } } peers { permanent_uuid: "1cf1378f5e8346fdb6f6873acc9b36fd" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42807 } } peers { permanent_uuid: "ea0f102d3dac4e259cfa59903189000c" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 34811 } }
I20260806 04:56:15.277551 1955 heartbeater.cc:503] Master 127.13.148.190:35709 was elected leader, sending a full tablet report...
I20260806 04:56:15.277961 2242 ts_tablet_manager.cc:1428] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260806 04:56:15.278730 2220 heartbeater.cc:503] Master 127.13.148.190:35709 was elected leader, sending a full tablet report...
I20260806 04:56:15.279376 2243 ts_tablet_manager.cc:1428] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260806 04:56:15.314751 1956 tablet.cc:2369] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:56:15.347682 2246 raft_consensus.cc:482] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:56:15.347905 2246 raft_consensus.cc:504] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44313 } } peers { permanent_uuid: "1cf1378f5e8346fdb6f6873acc9b36fd" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42807 } } peers { permanent_uuid: "ea0f102d3dac4e259cfa59903189000c" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 34811 } }
I20260806 04:56:15.349411 2246 leader_election.cc:288] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cf1378f5e8346fdb6f6873acc9b36fd (127.13.148.129:42807), ea0f102d3dac4e259cfa59903189000c (127.13.148.131:34811)
I20260806 04:56:15.350489 2247 raft_consensus.cc:482] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:56:15.350775 2247 raft_consensus.cc:504] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44313 } } peers { permanent_uuid: "1cf1378f5e8346fdb6f6873acc9b36fd" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42807 } } peers { permanent_uuid: "ea0f102d3dac4e259cfa59903189000c" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 34811 } }
I20260806 04:56:15.352531 2247 leader_election.cc:288] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3d5e14c06e46e4bd9a7c2da38ecb0c (127.13.148.130:44313), ea0f102d3dac4e259cfa59903189000c (127.13.148.131:34811)
I20260806 04:56:15.357805 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.350620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:36367 (local address 127.13.148.131:34811)
0806 04:56:15.350796 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:15.350801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:15.351373 (+ 572us) server_negotiation.cc:407] Connection header received
0806 04:56:15.351631 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:15.351638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:15.351707 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:15.351781 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:15.352230 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.353145 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.355651 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.356902 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.357040 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.357110 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:15.357466 (+ 356us) server_negotiation.cc:299] Negotiation successful
0806 04:56:15.357588 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:56:15.357919 2250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.350493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:34811 (local address 127.13.148.130:36367)
0806 04:56:15.351241 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:15.351259 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:15.351470 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:15.351808 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:15.351811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:15.351829 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:15.352088 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:15.352095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.353299 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.353305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:15.355411 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:15.355425 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.357160 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.357165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:15.357280 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.357301 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 04:56:15.357406 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 04:56:15.357529 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":82,"spinlock_wait_cycles":18176,"thread_start_us":423,"threads_started":1}
I20260806 04:56:15.358732 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.349735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.130:42687 (local address 127.13.148.129:42807)
0806 04:56:15.349927 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:15.349931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:15.350477 (+ 546us) server_negotiation.cc:407] Connection header received
0806 04:56:15.350784 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:15.350792 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:15.350905 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:15.350972 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:15.351792 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.352767 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.355682 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.357280 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.357464 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.357559 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:15.358169 (+ 610us) server_negotiation.cc:299] Negotiation successful
0806 04:56:15.358324 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":55}
I20260806 04:56:15.359269 2174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd557641624434192105cb3c024dfb0" candidate_uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0f102d3dac4e259cfa59903189000c" is_pre_election: true
I20260806 04:56:15.359607 2249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.349703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:42807 (local address 127.13.148.130:42687)
0806 04:56:15.350384 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:15.350403 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:15.350585 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:15.351188 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:15.351194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:15.351224 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:15.351656 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:15.351667 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.352941 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.352946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:15.355369 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:15.355397 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.357742 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.357750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:15.357983 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.358025 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 04:56:15.358969 (+ 944us) client_negotiation.cc:240] Negotiation successful
0806 04:56:15.359144 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":4,"thread_start_us":345,"threads_started":1}
I20260806 04:56:15.359901 2174 raft_consensus.cc:2454] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3d5e14c06e46e4bd9a7c2da38ecb0c in term 0.
I20260806 04:56:15.361063 1909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd557641624434192105cb3c024dfb0" candidate_uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cf1378f5e8346fdb6f6873acc9b36fd" is_pre_election: true
I20260806 04:56:15.361557 1909 raft_consensus.cc:2454] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3d5e14c06e46e4bd9a7c2da38ecb0c in term 0.
I20260806 04:56:15.361698 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.353339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:44391 (local address 127.13.148.130:44313)
0806 04:56:15.353829 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:15.353835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:15.353856 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:56:15.353900 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:15.353906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:15.353978 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:15.354081 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:15.354756 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.355728 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.359395 (+ 3667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.360571 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.360707 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.360794 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:15.361292 (+ 498us) server_negotiation.cc:299] Negotiation successful
0806 04:56:15.361437 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363}
I20260806 04:56:15.361860 1975 leader_election.cc:302] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df3d5e14c06e46e4bd9a7c2da38ecb0c, ea0f102d3dac4e259cfa59903189000c; no voters:
I20260806 04:56:15.362221 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.352854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.130:44313 (local address 127.13.148.129:44391)
0806 04:56:15.353613 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:15.353630 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:15.353729 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:15.354250 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:15.354255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:15.354279 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:15.354605 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:15.354615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.355898 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.355902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:15.359171 (+ 3269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:15.359186 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.360917 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.360922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:15.361116 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.361150 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 04:56:15.361256 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 04:56:15.361413 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":1091,"thread_start_us":1297,"threads_started":1}
I20260806 04:56:15.362247 2246 raft_consensus.cc:2781] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:56:15.362385 2246 raft_consensus.cc:482] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:56:15.362488 2246 raft_consensus.cc:3037] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:15.362958 2042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd557641624434192105cb3c024dfb0" candidate_uuid: "1cf1378f5e8346fdb6f6873acc9b36fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c" is_pre_election: true
I20260806 04:56:15.365324 2246 raft_consensus.cc:504] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44313 } } peers { permanent_uuid: "1cf1378f5e8346fdb6f6873acc9b36fd" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42807 } } peers { permanent_uuid: "ea0f102d3dac4e259cfa59903189000c" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 34811 } }
I20260806 04:56:15.365780 2042 raft_consensus.cc:2379] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cf1378f5e8346fdb6f6873acc9b36fd in current term 1: Already voted for candidate df3d5e14c06e46e4bd9a7c2da38ecb0c in this term.
I20260806 04:56:15.366017 2246 leader_election.cc:288] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [CANDIDATE]: Term 1 election: Requested vote from peers 1cf1378f5e8346fdb6f6873acc9b36fd (127.13.148.129:42807), ea0f102d3dac4e259cfa59903189000c (127.13.148.131:34811)
I20260806 04:56:15.366292 1909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd557641624434192105cb3c024dfb0" candidate_uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cf1378f5e8346fdb6f6873acc9b36fd"
I20260806 04:56:15.366466 1909 raft_consensus.cc:3037] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:15.366750 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.354849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.129:39397 (local address 127.13.148.131:34811)
0806 04:56:15.358025 (+ 3176us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:15.358032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:15.360671 (+ 2639us) server_negotiation.cc:407] Connection header received
0806 04:56:15.360918 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:15.360925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:15.360984 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:15.361036 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:56:15.361824 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.362746 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.364824 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.365508 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.365680 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.365749 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:15.366271 (+ 522us) server_negotiation.cc:299] Negotiation successful
0806 04:56:15.366406 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3094,"thread_start_us":5074,"threads_started":1}
I20260806 04:56:15.366858 2174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd557641624434192105cb3c024dfb0" candidate_uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0f102d3dac4e259cfa59903189000c"
I20260806 04:56:15.367051 2174 raft_consensus.cc:3037] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:15.367810 2252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.354551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.131:34811 (local address 127.13.148.129:39397)
0806 04:56:15.360598 (+ 6047us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:15.360614 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:15.360769 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:15.361159 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:15.361165 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:15.361181 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:56:15.361627 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:15.361639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.362864 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.362870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:15.364629 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:15.364642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.365703 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.365709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:15.366130 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.366164 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 04:56:15.366251 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 04:56:15.366370 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5929,"thread_start_us":5768,"threads_started":1}
I20260806 04:56:15.368440 2173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd557641624434192105cb3c024dfb0" candidate_uuid: "1cf1378f5e8346fdb6f6873acc9b36fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0f102d3dac4e259cfa59903189000c" is_pre_election: true
I20260806 04:56:15.369253 1843 leader_election.cc:302] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cf1378f5e8346fdb6f6873acc9b36fd; no voters: df3d5e14c06e46e4bd9a7c2da38ecb0c, ea0f102d3dac4e259cfa59903189000c
I20260806 04:56:15.369562 1909 raft_consensus.cc:2454] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3d5e14c06e46e4bd9a7c2da38ecb0c in term 1.
I20260806 04:56:15.369897 2247 raft_consensus.cc:2726] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:56:15.370055 1975 leader_election.cc:302] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf1378f5e8346fdb6f6873acc9b36fd, df3d5e14c06e46e4bd9a7c2da38ecb0c; no voters:
I20260806 04:56:15.370232 2174 raft_consensus.cc:2454] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3d5e14c06e46e4bd9a7c2da38ecb0c in term 1.
I20260806 04:56:15.370370 2246 raft_consensus.cc:2781] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:15.370837 2246 raft_consensus.cc:686] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [term 1 LEADER]: Becoming Leader. State: Replica: df3d5e14c06e46e4bd9a7c2da38ecb0c, State: Running, Role: LEADER
I20260806 04:56:15.371279 2246 consensus_queue.cc:237] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44313 } } peers { permanent_uuid: "1cf1378f5e8346fdb6f6873acc9b36fd" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42807 } } peers { permanent_uuid: "ea0f102d3dac4e259cfa59903189000c" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 34811 } }
I20260806 04:56:15.375295 1765 catalog_manager.cc:5495] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c reported cstate change: term changed from 0 to 1, leader changed from <none> to df3d5e14c06e46e4bd9a7c2da38ecb0c (127.13.148.130). New cstate: current_term: 1 leader_uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3d5e14c06e46e4bd9a7c2da38ecb0c" member_type: VOTER last_known_addr { host: "127.13.148.130" port: 44313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cf1378f5e8346fdb6f6873acc9b36fd" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0f102d3dac4e259cfa59903189000c" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 34811 } health_report { overall_health: UNKNOWN } } }
I20260806 04:56:15.393620 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.388844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44610 (local address 127.13.148.130:44313)
0806 04:56:15.389010 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:15.389014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:15.389057 (+ 43us) server_negotiation.cc:407] Connection header received
0806 04:56:15.389229 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:15.389233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:15.389329 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:15.389451 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:15.389767 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.390500 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.391856 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.392197 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.392277 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.392673 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:15.393184 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:15.393290 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:15.393384 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 04:56:15.393436 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 04:56:15.400914 2174 raft_consensus.cc:1261] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c [term 1 FOLLOWER]: Refusing update from remote peer df3d5e14c06e46e4bd9a7c2da38ecb0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:56:15.400964 1909 raft_consensus.cc:1261] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd [term 1 FOLLOWER]: Refusing update from remote peer df3d5e14c06e46e4bd9a7c2da38ecb0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:56:15.401736 2254 consensus_queue.cc:1035] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0f102d3dac4e259cfa59903189000c" member_type: VOTER last_known_addr { host: "127.13.148.131" port: 34811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:56:15.402096 2246 consensus_queue.cc:1035] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cf1378f5e8346fdb6f6873acc9b36fd" member_type: VOTER last_known_addr { host: "127.13.148.129" port: 42807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:56:15.413290 2258 mvcc.cc:205] Tried to move back new op lower bound from 7315423950432878592 to 7315423950324215808. Current Snapshot: MvccSnapshot[applied={T|T < 7315423950432878592}]
I20260806 04:56:15.416630 2257 mvcc.cc:205] Tried to move back new op lower bound from 7315423950432878592 to 7315423950324215808. Current Snapshot: MvccSnapshot[applied={T|T < 7315423950432878592 or (T in {7315423950432878592})}]
I20260806 04:56:15.424568 1764 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:56:15.424889 1764 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:56:15.428414 1764 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7284429e65e4d7b9655fa3a08494892: Sending DeleteTablet for 3 replicas of tablet 5dd557641624434192105cb3c024dfb0
I20260806 04:56:15.429458 2021 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd557641624434192105cb3c024dfb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:56:15 UTC) from {username='slave'} at 127.0.0.1:44602
I20260806 04:56:15.429862 1889 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd557641624434192105cb3c024dfb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:56:15 UTC) from {username='slave'} at 127.0.0.1:59910
I20260806 04:56:15.430042 2154 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd557641624434192105cb3c024dfb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:56:15 UTC) from {username='slave'} at 127.0.0.1:60986
I20260806 04:56:15.434993 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 1827
I20260806 04:56:15.440889 2270 tablet_replica.cc:330] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c: stopping tablet replica
I20260806 04:56:15.441399 2270 raft_consensus.cc:2229] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:56:15.442004 2270 raft_consensus.cc:2258] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:56:15.442814 2268 tablet_replica.cc:330] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c: stopping tablet replica
I20260806 04:56:15.443310 2269 tablet_replica.cc:330] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd: stopping tablet replica
I20260806 04:56:15.443943 2269 raft_consensus.cc:2229] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:56:15.445245 2270 ts_tablet_manager.cc:1905] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:56:15.445662 2268 raft_consensus.cc:2229] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:56:15.446234 2268 raft_consensus.cc:2258] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:56:15.446848 2269 raft_consensus.cc:2258] T 5dd557641624434192105cb3c024dfb0 P 1cf1378f5e8346fdb6f6873acc9b36fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:56:15.448571 2268 ts_tablet_manager.cc:1905] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:56:15.449733 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.436449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59912 (local address 127.13.148.129:42807)
0806 04:56:15.436680 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:15.436690 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:15.442971 (+ 6281us) server_negotiation.cc:407] Connection header received
0806 04:56:15.443188 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:15.443193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:15.443279 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:15.443378 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:15.443787 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.444711 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.446137 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.446689 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.446801 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.448072 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:15.448085 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:15.448100 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:15.448828 (+ 728us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:15.448897 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:15.448923 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:15.449018 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:15.449271 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 04:56:15.449394 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 04:56:15.450724 1950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:15.458148 2268 ts_tablet_manager.cc:1918] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:56:15.458333 2270 ts_tablet_manager.cc:1918] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:56:15.458462 2268 log.cc:1196] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-1/wal/wals/5dd557641624434192105cb3c024dfb0
I20260806 04:56:15.458563 2270 log.cc:1196] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c: Deleting WAL directory at /tmp/dist-test-task10hk37/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991872418975-13906-0/minicluster-data/ts-2/wal/wals/5dd557641624434192105cb3c024dfb0
I20260806 04:56:15.459110 2268 ts_tablet_manager.cc:1939] T 5dd557641624434192105cb3c024dfb0 P df3d5e14c06e46e4bd9a7c2da38ecb0c: Deleting consensus metadata
I20260806 04:56:15.459240 2270 ts_tablet_manager.cc:1939] T 5dd557641624434192105cb3c024dfb0 P ea0f102d3dac4e259cfa59903189000c: Deleting consensus metadata
I20260806 04:56:15.460992 1749 catalog_manager.cc:4841] TS ea0f102d3dac4e259cfa59903189000c (127.13.148.131:34811): tablet 5dd557641624434192105cb3c024dfb0 (table test-table [id=b3ebeb0b22e443eaa8459ecf2df6720a]) successfully deleted
I20260806 04:56:15.461313 1749 catalog_manager.cc:4841] TS df3d5e14c06e46e4bd9a7c2da38ecb0c (127.13.148.130:44313): tablet 5dd557641624434192105cb3c024dfb0 (table test-table [id=b3ebeb0b22e443eaa8459ecf2df6720a]) successfully deleted
I20260806 04:56:15.535569 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 1827
I20260806 04:56:15.560313 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 1959
W20260806 04:56:15.560675 1749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.129:42807 (error 108)
I20260806 04:56:15.563086 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.562569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:42807 (local address 127.0.0.1:59928)
0806 04:56:15.562775 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:15.562900 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:42807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260806 04:56:15.563319 1749 catalog_manager.cc:4584] TS 1cf1378f5e8346fdb6f6873acc9b36fd (127.13.148.129:42807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5dd557641624434192105cb3c024dfb0: Network error: Client connection negotiation failed: client connection to 127.13.148.129:42807: connect: Connection refused (error 111)
I20260806 04:56:15.567682 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.561475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.13.148.130:44313)
0806 04:56:15.561656 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:15.561678 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:15.561696 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:56:15.561770 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:15.561774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:15.561906 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:15.562028 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:15.562491 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.563299 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.564959 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.565334 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.565414 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.566422 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:15.566429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:15.566441 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:15.567009 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:15.567058 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:15.567076 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:15.567147 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:15.567297 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 04:56:15.567369 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 04:56:15.568341 2083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:15.616900 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.616365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:42807 (local address 127.0.0.1:59940)
0806 04:56:15.616551 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:15.616679 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:42807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 04:56:15.617883 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.617555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:42807 (local address 127.0.0.1:59950)
0806 04:56:15.617680 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:15.617745 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:42807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 04:56:15.669852 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 1959
I20260806 04:56:15.678424 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.678044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:42807 (local address 127.0.0.1:59956)
0806 04:56:15.678221 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:15.678294 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:42807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 04:56:15.679410 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.679051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:42807 (local address 127.0.0.1:59968)
0806 04:56:15.679207 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:15.679281 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:42807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 04:56:15.688526 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 2091
I20260806 04:56:15.698486 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.689318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32770 (local address 127.13.148.131:34811)
0806 04:56:15.689485 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:15.689489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:15.689507 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:56:15.689662 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:15.689667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:15.689758 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:15.689851 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:15.690202 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.690827 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.691864 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:15.692208 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:15.692305 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:15.697351 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:15.697360 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:15.697375 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:15.697944 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:15.697985 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:15.698008 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:15.698097 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:15.698247 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 04:56:15.698313 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:56:15.699088 2215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:15.759706 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.759252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:42807 (local address 127.0.0.1:59970)
0806 04:56:15.759465 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:15.759560 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:42807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 04:56:15.760771 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:15.760423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.129:42807 (local address 127.0.0.1:59972)
0806 04:56:15.760596 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:15.760675 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.129:42807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260806 04:56:15.780622 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 2091
I20260806 04:56:15.802977 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 1733
I20260806 04:56:15.803982 1797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:15.901942 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 1733
2026-08-06T04:56:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2597 ms)
[----------] 6 tests from EncryptionPolicyTest (20494 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:16 dist-test-slave-1x4g krb5kdc[2281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:16 dist-test-slave-1x4g krb5kdc[2281](info): set up 2 sockets
Aug 06 04:56:16 dist-test-slave-1x4g krb5kdc[2281](info): commencing operation
krb5kdc: starting...
W20260806 04:56:16.933136 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:17 dist-test-slave-1x4g krb5kdc[2281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:56:17.161196 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:17.461469 2294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:17.461716 2294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:17.461802 2294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:17.467907 2294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:17.467998 2294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:17.468037 2294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:17.468091 2294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:17.468128 2294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:17.476644 2294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41625
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:17.478318 2294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:17.479585 2294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:17.490463 2299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:17.490496 2300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:17.490463 2302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:17.491886 2294 server_base.cc:1034] running on GCE node
I20260806 04:56:17.492527 2294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:17.493772 2294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:17.494992 2294 hybrid_clock.cc:648] HybridClock initialized: now 1785992177494957 us; error 45 us; skew 500 ppm
I20260806 04:56:17.495299 2294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:17 dist-test-slave-1x4g krb5kdc[2281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:17.498294 2294 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:56:17.499641 2294 webserver.cc:460] Webserver started at http://127.0.0.1:37467/ using document root <none> and password file <none>
I20260806 04:56:17.500286 2294 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:17.500423 2294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:17.500697 2294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:17.503402 2294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "91e97706abf2463ca6ef167a950b0180"
format_stamp: "Formatted at 2026-08-06 04:56:17 on dist-test-slave-1x4g"
server_key: "36ca0661bc2ee3a7abfc53964fe2805c"
server_key_iv: "c97adfe3c6ceabb0024a8e4bb26bf02d"
server_key_version: "encryptionkey@0"
I20260806 04:56:17.504105 2294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "91e97706abf2463ca6ef167a950b0180"
format_stamp: "Formatted at 2026-08-06 04:56:17 on dist-test-slave-1x4g"
server_key: "36ca0661bc2ee3a7abfc53964fe2805c"
server_key_iv: "c97adfe3c6ceabb0024a8e4bb26bf02d"
server_key_version: "encryptionkey@0"
I20260806 04:56:17.508728 2294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260806 04:56:17.511982 2309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:17.513192 2294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:56:17.513372 2294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "91e97706abf2463ca6ef167a950b0180"
format_stamp: "Formatted at 2026-08-06 04:56:17 on dist-test-slave-1x4g"
server_key: "36ca0661bc2ee3a7abfc53964fe2805c"
server_key_iv: "c97adfe3c6ceabb0024a8e4bb26bf02d"
server_key_version: "encryptionkey@0"
I20260806 04:56:17.513546 2294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:17.541991 2294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:17.545746 2294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:17.546068 2294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:17.564118 2294 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41625
I20260806 04:56:17.564134 2362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41625 every 8 connection(s)
I20260806 04:56:17.565438 2294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:17.568626 2363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:17.572165 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 2294
I20260806 04:56:17.572364 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:17.573114 13906 external_mini_cluster.cc:1428] Setting key 1ce02c4b9604c98d81d679bc65c8aa76
I20260806 04:56:17.581236 2363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180: Bootstrap starting.
I20260806 04:56:17.584491 2363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:17.585671 2363 log.cc:824] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:56:17 dist-test-slave-1x4g krb5kdc[2281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 04:56:17.588346 2363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180: No bootstrap required, opened a new log
I20260806 04:56:17.595283 2366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:17.575043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35727 (local address 127.0.0.1:41625)
0806 04:56:17.575769 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:17.575780 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:17.575819 (+ 39us) server_negotiation.cc:407] Connection header received
0806 04:56:17.577471 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:17.577494 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:17.577900 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:17.578203 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:17.579935 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:17.581189 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:17.582747 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:17.583345 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:17.583473 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:17.587909 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:17.587916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:17.587928 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:17.587960 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:17.591278 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:17.592200 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:17.592205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:17.592210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:17.592369 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:17.592877 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:17.592880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:17.592882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:17.593366 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:17.593607 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:17.594044 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 04:56:17.594332 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":349,"threads_started":1}
I20260806 04:56:17.596390 2363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91e97706abf2463ca6ef167a950b0180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41625 } }
I20260806 04:56:17.596855 2363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:17.596980 2363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91e97706abf2463ca6ef167a950b0180, State: Initialized, Role: FOLLOWER
I20260806 04:56:17.597538 2363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91e97706abf2463ca6ef167a950b0180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41625 } }
I20260806 04:56:17.597854 2363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:17.597980 2363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:17.598151 2363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:17.599203 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:17.600986 2363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91e97706abf2463ca6ef167a950b0180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41625 } }
I20260806 04:56:17.601454 2363 leader_election.cc:302] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91e97706abf2463ca6ef167a950b0180; no voters:
I20260806 04:56:17.602075 2363 leader_election.cc:288] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:17.602135 2368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:17.603552 2368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [term 1 LEADER]: Becoming Leader. State: Replica: 91e97706abf2463ca6ef167a950b0180, State: Running, Role: LEADER
I20260806 04:56:17.604094 2368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91e97706abf2463ca6ef167a950b0180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41625 } }
I20260806 04:56:17.604244 2363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:17.606495 2371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91e97706abf2463ca6ef167a950b0180. Latest consensus state: current_term: 1 leader_uuid: "91e97706abf2463ca6ef167a950b0180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91e97706abf2463ca6ef167a950b0180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41625 } } }
I20260806 04:56:17.606868 2371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:17.607179 2369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91e97706abf2463ca6ef167a950b0180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91e97706abf2463ca6ef167a950b0180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41625 } } }
I20260806 04:56:17.607504 2369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:17.613936 2378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:17.618968 2378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:17.628041 2378 catalog_manager.cc:1347] Generated new cluster ID: a7138cf920a847869a27d11b0d7405e5
I20260806 04:56:17.628126 2378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:17.634318 2366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:17.608256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43299 (local address 127.0.0.1:41625)
0806 04:56:17.608391 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:17.608396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:17.615768 (+ 7372us) server_negotiation.cc:407] Connection header received
0806 04:56:17.615926 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:17.615934 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:17.616036 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:17.616168 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:17.618998 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:17.620642 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:17.622734 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:17.623285 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:17.623375 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:17.626448 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:17.626456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:17.626461 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:17.626499 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:17.630402 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:17.631189 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:17.631193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:17.631197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:17.631309 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:17.632757 (+ 1448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:17.632761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:17.632764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:17.633167 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:17.633371 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:17.633873 (+ 502us) server_negotiation.cc:299] Negotiation successful
0806 04:56:17.634027 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 04:56:17.647586 2378 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:17.648658 2378 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:17.654810 2378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91e97706abf2463ca6ef167a950b0180: Generated new TSK 0
I20260806 04:56:17.655476 2378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:17.664033 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 2294
I20260806 04:56:17.665278 2358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:17.745702 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 2294
W20260806 04:56:17.764068 2278 connection.cc:535] client connection to 127.0.0.1:41625 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41625): Connection reset by peer (error 104)
2026-08-06T04:56:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1834 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:17 dist-test-slave-1x4g krb5kdc[2396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:17 dist-test-slave-1x4g krb5kdc[2396](info): set up 2 sockets
Aug 06 04:56:17 dist-test-slave-1x4g krb5kdc[2396](info): commencing operation
krb5kdc: starting...
W20260806 04:56:18.756618 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:18 dist-test-slave-1x4g krb5kdc[2396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:56:18.989521 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:19.290629 2409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:19.290920 2409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:19.291083 2409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:19.297418 2409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:19.297565 2409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:19.297669 2409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:19.297787 2409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:19.297881 2409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:19.306403 2409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:19.307536 2409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:19.308741 2409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:19.319700 2415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:19.319737 2416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:19.319793 2419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:19.320524 2409 server_base.cc:1034] running on GCE node
I20260806 04:56:19.321276 2409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:19.322531 2409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:19.323731 2409 hybrid_clock.cc:648] HybridClock initialized: now 1785992179323689 us; error 30 us; skew 500 ppm
I20260806 04:56:19.324054 2409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:19 dist-test-slave-1x4g krb5kdc[2396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:19.326954 2409 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:56:19.328305 2409 webserver.cc:460] Webserver started at http://127.0.0.1:36825/ using document root <none> and password file <none>
I20260806 04:56:19.328996 2409 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:19.329150 2409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:19.329453 2409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:19.332158 2409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "0e40136e28f84e5ba567d9c692c7311d"
format_stamp: "Formatted at 2026-08-06 04:56:19 on dist-test-slave-1x4g"
server_key: "5c504531704feabfefe8ddcc84eb8527"
server_key_iv: "f2255d0a31067fb2f78916efa1982d77"
server_key_version: "encryptionkey@0"
I20260806 04:56:19.332911 2409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "0e40136e28f84e5ba567d9c692c7311d"
format_stamp: "Formatted at 2026-08-06 04:56:19 on dist-test-slave-1x4g"
server_key: "5c504531704feabfefe8ddcc84eb8527"
server_key_iv: "f2255d0a31067fb2f78916efa1982d77"
server_key_version: "encryptionkey@0"
I20260806 04:56:19.337328 2409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 04:56:19.340536 2426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:19.341809 2409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:56:19.341988 2409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "0e40136e28f84e5ba567d9c692c7311d"
format_stamp: "Formatted at 2026-08-06 04:56:19 on dist-test-slave-1x4g"
server_key: "5c504531704feabfefe8ddcc84eb8527"
server_key_iv: "f2255d0a31067fb2f78916efa1982d77"
server_key_version: "encryptionkey@0"
I20260806 04:56:19.342170 2409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:19.353601 2409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:19.356878 2409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:19.357153 2409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:19.374045 2409 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32973
I20260806 04:56:19.374076 2478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32973 every 8 connection(s)
I20260806 04:56:19.375248 2409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:19.378572 2479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:19.381582 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 2409
I20260806 04:56:19.381744 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:19.382236 13906 external_mini_cluster.cc:1428] Setting key 767a6f1b5a65c095c5c2f7e6aec1af0d
I20260806 04:56:19.390975 2479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d: Bootstrap starting.
I20260806 04:56:19.394550 2479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:19.395749 2479 log.cc:824] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:19.396543 2482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:19.383765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46499 (local address 127.0.0.1:32973)
0806 04:56:19.384427 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:19.384436 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:19.384471 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:56:19.386653 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:19.386679 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:19.387134 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:19.387578 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:19.388421 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:19.390417 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:19.391801 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:19.392795 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:19.392967 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:19.393995 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:19.394006 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:19.394023 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:19.394727 (+ 704us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:19.394810 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:19.394857 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:19.395021 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:19.395197 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 04:56:19.395333 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":313,"threads_started":1}
I20260806 04:56:19.398478 2479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d: No bootstrap required, opened a new log
I20260806 04:56:19.400077 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:19.405931 2479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e40136e28f84e5ba567d9c692c7311d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32973 } }
I20260806 04:56:19.406267 2479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:19.406371 2479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e40136e28f84e5ba567d9c692c7311d, State: Initialized, Role: FOLLOWER
I20260806 04:56:19.406927 2479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e40136e28f84e5ba567d9c692c7311d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32973 } }
I20260806 04:56:19.407184 2479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:19.407299 2479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:19.407430 2479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:19.410112 2482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:19.404232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55977 (local address 127.0.0.1:32973)
0806 04:56:19.404397 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:19.404405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:19.404709 (+ 304us) server_negotiation.cc:407] Connection header received
0806 04:56:19.404884 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:19.404887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:19.404935 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:19.405001 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:19.405523 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:19.406127 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:19.407740 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:19.408125 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:19.408210 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:19.409386 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:19.409390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:19.409393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:19.409409 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:19.409442 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:19.409455 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:19.409534 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:19.409733 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 04:56:19.409795 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:56:19.410274 2479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e40136e28f84e5ba567d9c692c7311d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32973 } }
I20260806 04:56:19.410830 2479 leader_election.cc:302] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e40136e28f84e5ba567d9c692c7311d; no voters:
I20260806 04:56:19.411476 2479 leader_election.cc:288] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:19.411593 2489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:19.412259 2489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [term 1 LEADER]: Becoming Leader. State: Replica: 0e40136e28f84e5ba567d9c692c7311d, State: Running, Role: LEADER
I20260806 04:56:19.412837 2489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e40136e28f84e5ba567d9c692c7311d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32973 } }
I20260806 04:56:19.413558 2479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:19.415716 2491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e40136e28f84e5ba567d9c692c7311d. Latest consensus state: current_term: 1 leader_uuid: "0e40136e28f84e5ba567d9c692c7311d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e40136e28f84e5ba567d9c692c7311d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32973 } } }
I20260806 04:56:19.415596 2490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e40136e28f84e5ba567d9c692c7311d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e40136e28f84e5ba567d9c692c7311d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32973 } } }
I20260806 04:56:19.416608 2490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:19.417122 2491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:19.421880 2495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:19.427594 2495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:19.435428 2495 catalog_manager.cc:1347] Generated new cluster ID: 1e200e56f8fd4ea2ba24cfcd30fec80e
I20260806 04:56:19.435539 2495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:19.452559 2495 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:19.453719 2495 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:19.468622 2495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e40136e28f84e5ba567d9c692c7311d: Generated new TSK 0
I20260806 04:56:19.469385 2495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:19.483306 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 2409
W20260806 04:56:19.484344 2474 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46499
W20260806 04:56:19.485050 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 2409: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:56:19.485213 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 2409
W20260806 04:56:19.503346 2394 connection.cc:535] client connection to 127.0.0.1:32973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32973): Connection reset by peer (error 104)
2026-08-06T04:56:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1739 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:19 dist-test-slave-1x4g krb5kdc[2510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:19 dist-test-slave-1x4g krb5kdc[2510](info): set up 2 sockets
Aug 06 04:56:19 dist-test-slave-1x4g krb5kdc[2510](info): commencing operation
krb5kdc: starting...
W20260806 04:56:20.505925 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:20 dist-test-slave-1x4g krb5kdc[2510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:56:20.735561 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:21.044608 2523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:21.044956 2523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:21.045099 2523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:21.051419 2523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:21.051546 2523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:21.051633 2523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:21.051730 2523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:21.051803 2523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:21.060639 2523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41875
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:21.061782 2523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:21.062937 2523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:21.073360 2528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:21.073491 2532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:21.073606 2529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:21.074532 2523 server_base.cc:1034] running on GCE node
I20260806 04:56:21.075289 2523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:21.076552 2523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:21.077796 2523 hybrid_clock.cc:648] HybridClock initialized: now 1785992181077731 us; error 78 us; skew 500 ppm
I20260806 04:56:21.078138 2523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:21 dist-test-slave-1x4g krb5kdc[2510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:21.084903 2523 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:56:21.086277 2523 webserver.cc:460] Webserver started at http://127.0.0.1:36435/ using document root <none> and password file <none>
I20260806 04:56:21.086853 2523 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:21.086972 2523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:21.087252 2523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:21.089951 2523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "b3f8681477804779b54058e22655e932"
format_stamp: "Formatted at 2026-08-06 04:56:21 on dist-test-slave-1x4g"
server_key: "aae4819cab3756507f1e110ae184e44c"
server_key_iv: "453897ac77dcae77f33bc7850fc35e26"
server_key_version: "encryptionkey@0"
I20260806 04:56:21.090678 2523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "b3f8681477804779b54058e22655e932"
format_stamp: "Formatted at 2026-08-06 04:56:21 on dist-test-slave-1x4g"
server_key: "aae4819cab3756507f1e110ae184e44c"
server_key_iv: "453897ac77dcae77f33bc7850fc35e26"
server_key_version: "encryptionkey@0"
I20260806 04:56:21.095580 2523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 04:56:21.098872 2539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:21.100217 2523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 04:56:21.100404 2523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "b3f8681477804779b54058e22655e932"
format_stamp: "Formatted at 2026-08-06 04:56:21 on dist-test-slave-1x4g"
server_key: "aae4819cab3756507f1e110ae184e44c"
server_key_iv: "453897ac77dcae77f33bc7850fc35e26"
server_key_version: "encryptionkey@0"
I20260806 04:56:21.100585 2523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:21.105682 2523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:21.109594 2523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:21.109890 2523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:21.127080 2523 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41875
I20260806 04:56:21.127121 2591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41875 every 8 connection(s)
I20260806 04:56:21.128309 2523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:21.131855 2592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:21.136019 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 2523
I20260806 04:56:21.136175 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:21.136721 13906 external_mini_cluster.cc:1428] Setting key 80ceabb6811d7c7a55343b20cbaece66
I20260806 04:56:21.143410 2592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932: Bootstrap starting.
I20260806 04:56:21.145107 2596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:21.138406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36521 (local address 127.0.0.1:41875)
0806 04:56:21.139130 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:21.139151 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:21.139202 (+ 51us) server_negotiation.cc:407] Connection header received
0806 04:56:21.141783 (+ 2581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:21.141807 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:21.142175 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:21.142544 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:21.143426 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:21.143435 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:21.143450 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:21.143467 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:21.143532 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:21.143560 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:21.143752 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:21.143866 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 04:56:21.144033 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":289,"threads_started":1}
I20260806 04:56:21.146690 2592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:21.147810 2592 log.cc:824] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:21.148521 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:21.150381 2592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932: No bootstrap required, opened a new log
I20260806 04:56:21.154924 2596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:21.152712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49103 (local address 127.0.0.1:41875)
0806 04:56:21.152883 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:21.152889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:21.153320 (+ 431us) server_negotiation.cc:407] Connection header received
0806 04:56:21.153448 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:21.153452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:21.153492 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:21.153560 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:21.154308 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:21.154311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:21.154314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:21.154325 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:21.154356 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:21.154371 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:21.154451 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:21.154568 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 04:56:21.154605 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 04:56:21.158318 2592 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f8681477804779b54058e22655e932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41875 } }
I20260806 04:56:21.158638 2592 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:21.158731 2592 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f8681477804779b54058e22655e932, State: Initialized, Role: FOLLOWER
I20260806 04:56:21.159271 2592 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f8681477804779b54058e22655e932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41875 } }
I20260806 04:56:21.159531 2592 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:21.159688 2592 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:21.159896 2592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:21.161883 2592 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f8681477804779b54058e22655e932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41875 } }
I20260806 04:56:21.162313 2592 leader_election.cc:302] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3f8681477804779b54058e22655e932; no voters:
I20260806 04:56:21.162844 2592 leader_election.cc:288] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:21.163033 2603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:21.164315 2603 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [term 1 LEADER]: Becoming Leader. State: Replica: b3f8681477804779b54058e22655e932, State: Running, Role: LEADER
I20260806 04:56:21.164891 2592 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:21.164904 2603 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f8681477804779b54058e22655e932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41875 } }
I20260806 04:56:21.167924 2605 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f8681477804779b54058e22655e932. Latest consensus state: current_term: 1 leader_uuid: "b3f8681477804779b54058e22655e932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f8681477804779b54058e22655e932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41875 } } }
I20260806 04:56:21.168008 2604 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3f8681477804779b54058e22655e932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f8681477804779b54058e22655e932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41875 } } }
I20260806 04:56:21.168300 2605 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:21.168349 2604 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:21.169848 2612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:21.176129 2612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:21.184450 2612 catalog_manager.cc:1347] Generated new cluster ID: 6322cb8fb3884916961493624734c430
I20260806 04:56:21.184556 2612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:21.206053 2612 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:21.206624 2612 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:21.212936 2612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3f8681477804779b54058e22655e932: Generated new TSK 0
I20260806 04:56:21.213752 2612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:21.220425 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 2523
W20260806 04:56:21.221402 2587 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36521
W20260806 04:56:21.222036 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 2523: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:56:21.222158 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 2523
2026-08-06T04:56:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1737 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:21 dist-test-slave-1x4g krb5kdc[2625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:21 dist-test-slave-1x4g krb5kdc[2625](info): set up 2 sockets
Aug 06 04:56:21 dist-test-slave-1x4g krb5kdc[2625](info): commencing operation
krb5kdc: starting...
W20260806 04:56:22.228734 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:22 dist-test-slave-1x4g krb5kdc[2625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:56:22.470135 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:22.773633 2638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:22.773931 2638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:22.774075 2638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:22.780617 2638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:22.780750 2638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:22.780866 2638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:22.780967 2638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:22.781032 2638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:22.789592 2638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:22.790755 2638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:22.791956 2638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:22.802776 2643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:22.802913 2646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:22.802979 2644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:22.803795 2638 server_base.cc:1034] running on GCE node
I20260806 04:56:22.804531 2638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:22.805747 2638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:22.806972 2638 hybrid_clock.cc:648] HybridClock initialized: now 1785992182806933 us; error 41 us; skew 500 ppm
I20260806 04:56:22.807307 2638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:22 dist-test-slave-1x4g krb5kdc[2625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:22.810276 2638 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:56:22.811738 2638 webserver.cc:460] Webserver started at http://127.0.0.1:34337/ using document root <none> and password file <none>
I20260806 04:56:22.812345 2638 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:22.812484 2638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:22.812762 2638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:22.815519 2638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "d9f18b62943f4d51a5368c48829f2cb2"
format_stamp: "Formatted at 2026-08-06 04:56:22 on dist-test-slave-1x4g"
server_key: "d559b594c23ef68d394dd5bef0e0e84d"
server_key_iv: "58506be3831da2435e889b25ca67ebfe"
server_key_version: "encryptionkey@0"
I20260806 04:56:22.816246 2638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "d9f18b62943f4d51a5368c48829f2cb2"
format_stamp: "Formatted at 2026-08-06 04:56:22 on dist-test-slave-1x4g"
server_key: "d559b594c23ef68d394dd5bef0e0e84d"
server_key_iv: "58506be3831da2435e889b25ca67ebfe"
server_key_version: "encryptionkey@0"
I20260806 04:56:22.821282 2638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260806 04:56:22.824296 2653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:22.825518 2638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:56:22.825677 2638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "d9f18b62943f4d51a5368c48829f2cb2"
format_stamp: "Formatted at 2026-08-06 04:56:22 on dist-test-slave-1x4g"
server_key: "d559b594c23ef68d394dd5bef0e0e84d"
server_key_iv: "58506be3831da2435e889b25ca67ebfe"
server_key_version: "encryptionkey@0"
I20260806 04:56:22.825843 2638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:22.849347 2638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:22.852886 2638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:22.853140 2638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:22.870151 2638 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39977
I20260806 04:56:22.870174 2706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39977 every 8 connection(s)
I20260806 04:56:22.871371 2638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:22.871886 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 2638
I20260806 04:56:22.872069 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:22.872593 13906 external_mini_cluster.cc:1428] Setting key ff739fbee814dca71367ff94dacac267
I20260806 04:56:22.875628 2707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 04:56:22 dist-test-slave-1x4g krb5kdc[2625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 04:56:22.886905 2707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2: Bootstrap starting.
I20260806 04:56:22.889828 2707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:22.890823 2707 log.cc:824] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:22.893376 2707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2: No bootstrap required, opened a new log
I20260806 04:56:22.893867 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:22.874328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36973 (local address 127.0.0.1:39977)
0806 04:56:22.875067 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:22.875079 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:22.875125 (+ 46us) server_negotiation.cc:407] Connection header received
0806 04:56:22.877372 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:22.877397 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:22.877869 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:22.878268 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:22.879422 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:22.881350 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:22.882418 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:22.883315 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:22.883466 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:22.887233 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:22.887240 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:22.887252 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:22.887282 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:22.890179 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:22.890852 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:22.890858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:22.890863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:22.891060 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:22.891429 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:22.891434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:22.891437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:22.891910 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:22.892174 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:22.892585 (+ 411us) server_negotiation.cc:299] Negotiation successful
0806 04:56:22.892943 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":313,"threads_started":1}
I20260806 04:56:22.897485 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:22.901499 2707 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f18b62943f4d51a5368c48829f2cb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39977 } }
I20260806 04:56:22.901877 2707 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:22.901978 2707 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f18b62943f4d51a5368c48829f2cb2, State: Initialized, Role: FOLLOWER
I20260806 04:56:22.902585 2707 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f18b62943f4d51a5368c48829f2cb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39977 } }
I20260806 04:56:22.902832 2707 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:22.902966 2707 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:22.903120 2707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:22.905468 2707 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f18b62943f4d51a5368c48829f2cb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39977 } }
I20260806 04:56:22.905927 2707 leader_election.cc:302] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9f18b62943f4d51a5368c48829f2cb2; no voters:
I20260806 04:56:22.906589 2707 leader_election.cc:288] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:22.906713 2718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:22.908432 2707 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:22.909102 2718 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [term 1 LEADER]: Becoming Leader. State: Replica: d9f18b62943f4d51a5368c48829f2cb2, State: Running, Role: LEADER
I20260806 04:56:22.909806 2718 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f18b62943f4d51a5368c48829f2cb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39977 } }
I20260806 04:56:22.916942 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:22.901579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42467 (local address 127.0.0.1:39977)
0806 04:56:22.901738 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:22.901745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:22.902197 (+ 452us) server_negotiation.cc:407] Connection header received
0806 04:56:22.902349 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:22.902354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:22.902415 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:22.902507 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:22.903785 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:22.904737 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:22.905852 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:22.906527 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:22.906689 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:22.910116 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:22.910122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:22.910127 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:22.910162 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:22.914054 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:22.914821 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:22.914829 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:22.914831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:22.915033 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:22.915424 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:22.915429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:22.915433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:22.915786 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:22.916200 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:22.916333 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 04:56:22.916531 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 04:56:22.917248 2719 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9f18b62943f4d51a5368c48829f2cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f18b62943f4d51a5368c48829f2cb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39977 } } }
I20260806 04:56:22.917382 2720 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9f18b62943f4d51a5368c48829f2cb2. Latest consensus state: current_term: 1 leader_uuid: "d9f18b62943f4d51a5368c48829f2cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f18b62943f4d51a5368c48829f2cb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39977 } } }
I20260806 04:56:22.917662 2719 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:22.917686 2720 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2 [sys.catalog]: This master's current role is: LEADER
W20260806 04:56:22.924873 2734 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:56:22.925050 2734 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:56:22.925206 2733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:22.926671 2733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:22.934804 2733 catalog_manager.cc:1347] Generated new cluster ID: 66dcff575d2f4a4cba9ad66c859d30cc
I20260806 04:56:22.934965 2733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:22.943341 2733 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:22.944442 2733 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:22.960875 2733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9f18b62943f4d51a5368c48829f2cb2: Generated new TSK 0
I20260806 04:56:22.961829 2733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:22.976168 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 2638
I20260806 04:56:22.977507 2702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:23.061254 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 2638
2026-08-06T04:56:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1838 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:23 dist-test-slave-1x4g krb5kdc[2741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:23 dist-test-slave-1x4g krb5kdc[2741](info): set up 2 sockets
Aug 06 04:56:23 dist-test-slave-1x4g krb5kdc[2741](info): commencing operation
krb5kdc: starting...
W20260806 04:56:24.076956 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:24 dist-test-slave-1x4g krb5kdc[2741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:56:24.325789 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:24.634222 2754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:24.634543 2754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:24.634706 2754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:24.641637 2754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:24.641778 2754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:24.641896 2754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:24.642006 2754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:24.642079 2754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:24.651120 2754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:24.652311 2754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:24.653563 2754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:24.664976 2760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:24.665006 2762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:24.665413 2759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:24.665944 2754 server_base.cc:1034] running on GCE node
I20260806 04:56:24.666738 2754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:24.668172 2754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:24.669437 2754 hybrid_clock.cc:648] HybridClock initialized: now 1785992184669390 us; error 32 us; skew 500 ppm
I20260806 04:56:24.669802 2754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:24 dist-test-slave-1x4g krb5kdc[2741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:24.672883 2754 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:56:24.674165 2754 webserver.cc:460] Webserver started at http://127.0.0.1:43039/ using document root <none> and password file <none>
I20260806 04:56:24.674786 2754 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:24.674944 2754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:24.675238 2754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:24.678314 2754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "531f6486b1d349d3ac58b85e32c8393d"
format_stamp: "Formatted at 2026-08-06 04:56:24 on dist-test-slave-1x4g"
server_key: "4cefb0f07c7b138c1ebf701cc0fc36f0"
server_key_iv: "c45415fbbf731acc9360ad9f87df4b49"
server_key_version: "encryptionkey@0"
I20260806 04:56:24.679085 2754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "531f6486b1d349d3ac58b85e32c8393d"
format_stamp: "Formatted at 2026-08-06 04:56:24 on dist-test-slave-1x4g"
server_key: "4cefb0f07c7b138c1ebf701cc0fc36f0"
server_key_iv: "c45415fbbf731acc9360ad9f87df4b49"
server_key_version: "encryptionkey@0"
I20260806 04:56:24.684365 2754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 04:56:24.687609 2769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:24.688905 2754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:56:24.689077 2754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "531f6486b1d349d3ac58b85e32c8393d"
format_stamp: "Formatted at 2026-08-06 04:56:24 on dist-test-slave-1x4g"
server_key: "4cefb0f07c7b138c1ebf701cc0fc36f0"
server_key_iv: "c45415fbbf731acc9360ad9f87df4b49"
server_key_version: "encryptionkey@0"
I20260806 04:56:24.689253 2754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:24.703095 2754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:24.706573 2754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:24.706816 2754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:24.724053 2754 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33871
I20260806 04:56:24.724078 2822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33871 every 8 connection(s)
I20260806 04:56:24.725275 2754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:24.728395 2823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:24.728848 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 2754
I20260806 04:56:24.729018 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:24.729579 13906 external_mini_cluster.cc:1428] Setting key 66c59ada565139a634955a36ead61cda
I20260806 04:56:24.740474 2823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d: Bootstrap starting.
I20260806 04:56:24.741869 2826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:24.731187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54723 (local address 127.0.0.1:33871)
0806 04:56:24.731910 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:24.731919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:24.731956 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:56:24.733927 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:24.733945 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:24.734283 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:24.734571 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:24.735068 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:24.736115 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:24.737883 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:24.738482 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:24.738592 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:24.739685 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:24.739692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:24.739704 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:24.740243 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:24.740314 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:24.740338 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:24.740422 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:24.740748 (+ 326us) server_negotiation.cc:299] Negotiation successful
0806 04:56:24.741049 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":371,"threads_started":1}
I20260806 04:56:24.743285 2823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:24.744297 2823 log.cc:824] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:24.745038 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:24.746974 2823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d: No bootstrap required, opened a new log
I20260806 04:56:24.755617 2826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:24.750010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60701 (local address 127.0.0.1:33871)
0806 04:56:24.750191 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:24.750197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:24.750660 (+ 463us) server_negotiation.cc:407] Connection header received
0806 04:56:24.750775 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:24.750784 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:24.750850 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:24.750941 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:24.751355 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:24.752295 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:24.753443 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:24.754051 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:24.754155 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:24.755008 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:24.755013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:24.755017 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:24.755033 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:24.755069 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:24.755085 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:24.755165 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:24.755269 (+ 104us) server_negotiation.cc:299] Negotiation successful
0806 04:56:24.755339 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:56:24.755692 2823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531f6486b1d349d3ac58b85e32c8393d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33871 } }
I20260806 04:56:24.756038 2823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:24.756134 2823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 531f6486b1d349d3ac58b85e32c8393d, State: Initialized, Role: FOLLOWER
I20260806 04:56:24.756645 2823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531f6486b1d349d3ac58b85e32c8393d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33871 } }
I20260806 04:56:24.756968 2823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:24.757122 2823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:24.757295 2823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:24.759121 2823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531f6486b1d349d3ac58b85e32c8393d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33871 } }
I20260806 04:56:24.759599 2823 leader_election.cc:302] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 531f6486b1d349d3ac58b85e32c8393d; no voters:
I20260806 04:56:24.760142 2823 leader_election.cc:288] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:24.760236 2833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:24.761411 2833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [term 1 LEADER]: Becoming Leader. State: Replica: 531f6486b1d349d3ac58b85e32c8393d, State: Running, Role: LEADER
I20260806 04:56:24.761845 2833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531f6486b1d349d3ac58b85e32c8393d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33871 } }
I20260806 04:56:24.761974 2823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:24.764549 2835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 531f6486b1d349d3ac58b85e32c8393d. Latest consensus state: current_term: 1 leader_uuid: "531f6486b1d349d3ac58b85e32c8393d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531f6486b1d349d3ac58b85e32c8393d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33871 } } }
I20260806 04:56:24.764890 2835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:24.765578 2834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "531f6486b1d349d3ac58b85e32c8393d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531f6486b1d349d3ac58b85e32c8393d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33871 } } }
I20260806 04:56:24.765874 2834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:24.773350 2840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:24.778339 2840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:24.785586 2840 catalog_manager.cc:1347] Generated new cluster ID: d423f3d680d9410b963364df71d9ab59
I20260806 04:56:24.785679 2840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:24.807354 2840 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:24.808357 2840 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:24.814034 2840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 531f6486b1d349d3ac58b85e32c8393d: Generated new TSK 0
I20260806 04:56:24.814796 2840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:24.832054 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 2754
W20260806 04:56:24.833452 2817 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54723
W20260806 04:56:24.834158 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 2754: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:56:24.834322 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 2754
2026-08-06T04:56:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1773 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:24 dist-test-slave-1x4g krb5kdc[2855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:24 dist-test-slave-1x4g krb5kdc[2855](info): set up 2 sockets
Aug 06 04:56:24 dist-test-slave-1x4g krb5kdc[2855](info): commencing operation
krb5kdc: starting...
W20260806 04:56:25.844921 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:25 dist-test-slave-1x4g krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:56:26.077416 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36381
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:26.381726 2869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:26.382058 2869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:26.382232 2869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:26.388715 2869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:26.388875 2869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:26.388963 2869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:26.389077 2869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:26.389158 2869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:26.398597 2869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:26.399899 2869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:26.401275 2869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:26.412694 2875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:26.412736 2874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:26.412919 2877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:26.414126 2869 server_base.cc:1034] running on GCE node
I20260806 04:56:26.414778 2869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:26.416002 2869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:26.417213 2869 hybrid_clock.cc:648] HybridClock initialized: now 1785992186417160 us; error 48 us; skew 500 ppm
I20260806 04:56:26.417542 2869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:26 dist-test-slave-1x4g krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:26.424216 2869 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:56:26.425734 2869 webserver.cc:460] Webserver started at http://127.0.0.1:34891/ using document root <none> and password file <none>
I20260806 04:56:26.426393 2869 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:26.426542 2869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:26.426896 2869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:26.429867 2869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "835fcdcb51464891af46e76f30b86f02"
format_stamp: "Formatted at 2026-08-06 04:56:26 on dist-test-slave-1x4g"
server_key: "89adfde139db6a0d38caa32e907969ab"
server_key_iv: "d736cd09aa91911a044a9db25105b777"
server_key_version: "encryptionkey@0"
I20260806 04:56:26.430631 2869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "835fcdcb51464891af46e76f30b86f02"
format_stamp: "Formatted at 2026-08-06 04:56:26 on dist-test-slave-1x4g"
server_key: "89adfde139db6a0d38caa32e907969ab"
server_key_iv: "d736cd09aa91911a044a9db25105b777"
server_key_version: "encryptionkey@0"
I20260806 04:56:26.435617 2869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 04:56:26.438964 2884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:26.440204 2869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260806 04:56:26.440387 2869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "835fcdcb51464891af46e76f30b86f02"
format_stamp: "Formatted at 2026-08-06 04:56:26 on dist-test-slave-1x4g"
server_key: "89adfde139db6a0d38caa32e907969ab"
server_key_iv: "d736cd09aa91911a044a9db25105b777"
server_key_version: "encryptionkey@0"
I20260806 04:56:26.440604 2869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:26.446174 2869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:26.449870 2869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:26.450169 2869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:26.467836 2869 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36381
I20260806 04:56:26.467905 2936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36381 every 8 connection(s)
I20260806 04:56:26.469115 2869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:26.469409 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 2869
I20260806 04:56:26.469604 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:26.470108 13906 external_mini_cluster.cc:1428] Setting key a387d7cb13f1402712e08904ba534381
I20260806 04:56:26.473455 2937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:26.477535 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:26.471882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35029 (local address 127.0.0.1:36381)
0806 04:56:26.472611 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:26.472622 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:26.472659 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:56:26.474254 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:26.474275 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:26.474580 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:26.474930 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:26.475956 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:26.475961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:26.475972 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:26.475992 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:26.476057 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:26.476074 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:26.476307 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:26.476451 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 04:56:26.476632 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":320,"threads_started":1}
I20260806 04:56:26.481387 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:26.484584 2937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02: Bootstrap starting.
I20260806 04:56:26.487802 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:26.485468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49093 (local address 127.0.0.1:36381)
0806 04:56:26.485633 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:26.485639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:26.486103 (+ 464us) server_negotiation.cc:407] Connection header received
0806 04:56:26.486216 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:26.486221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:26.486285 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:26.486399 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:26.487224 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:26.487231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:26.487236 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:26.487255 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:26.487306 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:26.487326 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:26.487426 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:26.487507 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 04:56:26.487561 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:56:26.488178 2937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:26.489426 2937 log.cc:824] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:26.492172 2937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02: No bootstrap required, opened a new log
I20260806 04:56:26.499533 2937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "835fcdcb51464891af46e76f30b86f02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36381 } }
I20260806 04:56:26.499891 2937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:26.500010 2937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835fcdcb51464891af46e76f30b86f02, State: Initialized, Role: FOLLOWER
I20260806 04:56:26.500581 2937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "835fcdcb51464891af46e76f30b86f02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36381 } }
I20260806 04:56:26.500860 2937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:26.501008 2937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:26.501178 2937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:26.503520 2937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "835fcdcb51464891af46e76f30b86f02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36381 } }
I20260806 04:56:26.504050 2937 leader_election.cc:302] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 835fcdcb51464891af46e76f30b86f02; no voters:
I20260806 04:56:26.504595 2937 leader_election.cc:288] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:26.504698 2947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:26.505422 2947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [term 1 LEADER]: Becoming Leader. State: Replica: 835fcdcb51464891af46e76f30b86f02, State: Running, Role: LEADER
I20260806 04:56:26.505894 2947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "835fcdcb51464891af46e76f30b86f02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36381 } }
I20260806 04:56:26.506528 2937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:26.508980 2949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 835fcdcb51464891af46e76f30b86f02. Latest consensus state: current_term: 1 leader_uuid: "835fcdcb51464891af46e76f30b86f02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "835fcdcb51464891af46e76f30b86f02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36381 } } }
I20260806 04:56:26.509450 2949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:26.510392 2948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "835fcdcb51464891af46e76f30b86f02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "835fcdcb51464891af46e76f30b86f02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36381 } } }
I20260806 04:56:26.510851 2948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:26.512465 2956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:26.517653 2956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:26.526015 2956 catalog_manager.cc:1347] Generated new cluster ID: fd889d44111f4c55aa9ba69b9bc0bc19
I20260806 04:56:26.526118 2956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:26.547881 2956 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:26.548228 2956 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:26.554771 2956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 835fcdcb51464891af46e76f30b86f02: Generated new TSK 0
I20260806 04:56:26.555401 2956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:26.564316 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 2869
W20260806 04:56:26.565271 2932 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35029
W20260806 04:56:26.565945 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 2869: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:56:26.566084 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 2869
2026-08-06T04:56:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1732 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:26 dist-test-slave-1x4g krb5kdc[2968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:26 dist-test-slave-1x4g krb5kdc[2968](info): set up 2 sockets
Aug 06 04:56:26 dist-test-slave-1x4g krb5kdc[2968](info): commencing operation
krb5kdc: starting...
W20260806 04:56:27.580986 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:27 dist-test-slave-1x4g krb5kdc[2968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:56:27.807709 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:28.111922 2982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:28.112201 2982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:28.112344 2982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:28.118674 2982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:28.118791 2982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:28.118880 2982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:28.119011 2982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:28.119091 2982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:28.127807 2982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:28.128906 2982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:28.130115 2982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:28.140861 2988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:28.140937 2989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:28.140894 2991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:28.141645 2982 server_base.cc:1034] running on GCE node
I20260806 04:56:28.142325 2982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:28.143455 2982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:28.144639 2982 hybrid_clock.cc:648] HybridClock initialized: now 1785992188144607 us; error 27 us; skew 500 ppm
I20260806 04:56:28.144984 2982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:28 dist-test-slave-1x4g krb5kdc[2968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:28.147753 2982 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:56:28.148998 2982 webserver.cc:460] Webserver started at http://127.0.0.1:44053/ using document root <none> and password file <none>
I20260806 04:56:28.149533 2982 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:28.149650 2982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:28.149886 2982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:28.152465 2982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "907885f5a1c54d0a9707a29f836bb8b6"
format_stamp: "Formatted at 2026-08-06 04:56:28 on dist-test-slave-1x4g"
server_key: "101029a9d274514d942dbd8c7458da3d"
server_key_iv: "d0eb70074aa1e3da36622094adc43ee2"
server_key_version: "encryptionkey@0"
I20260806 04:56:28.153239 2982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "907885f5a1c54d0a9707a29f836bb8b6"
format_stamp: "Formatted at 2026-08-06 04:56:28 on dist-test-slave-1x4g"
server_key: "101029a9d274514d942dbd8c7458da3d"
server_key_iv: "d0eb70074aa1e3da36622094adc43ee2"
server_key_version: "encryptionkey@0"
I20260806 04:56:28.158047 2982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260806 04:56:28.161104 2998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:28.162271 2982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:56:28.162443 2982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "907885f5a1c54d0a9707a29f836bb8b6"
format_stamp: "Formatted at 2026-08-06 04:56:28 on dist-test-slave-1x4g"
server_key: "101029a9d274514d942dbd8c7458da3d"
server_key_iv: "d0eb70074aa1e3da36622094adc43ee2"
server_key_version: "encryptionkey@0"
I20260806 04:56:28.162631 2982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:28.177515 2982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:28.181169 2982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:28.181411 2982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:28.198179 2982 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46825
I20260806 04:56:28.198211 3049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46825 every 8 connection(s)
I20260806 04:56:28.199330 2982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:28.202867 3050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:28.207978 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 2982
I20260806 04:56:28.208160 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:28.208629 13906 external_mini_cluster.cc:1428] Setting key 3a3a0383f85e7b67be0797a65e72f017
I20260806 04:56:28.212922 3050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6: Bootstrap starting.
I20260806 04:56:28.216329 3050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:28.217613 3050 log.cc:824] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:28.220546 3050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6: No bootstrap required, opened a new log
Aug 06 04:56:28 dist-test-slave-1x4g krb5kdc[2968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 04:56:28.229756 3050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907885f5a1c54d0a9707a29f836bb8b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46825 } }
I20260806 04:56:28.230175 3050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:28.230301 3050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907885f5a1c54d0a9707a29f836bb8b6, State: Initialized, Role: FOLLOWER
I20260806 04:56:28.230937 3050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907885f5a1c54d0a9707a29f836bb8b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46825 } }
I20260806 04:56:28.231225 3050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:28.231374 3050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:28.231590 3050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:28.231630 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:28.210494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:59643 (local address 127.0.0.1:46825)
0806 04:56:28.211266 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:28.211276 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:28.211313 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:56:28.212903 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:28.212927 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:28.213322 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:28.213630 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:28.215369 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:28.216615 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:28.218410 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:28.219119 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:28.219230 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:28.223738 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:28.223751 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:28.223766 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:28.223807 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:28.227398 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:28.228043 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:28.228055 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:28.228079 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:28.228290 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:28.228668 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:28.228673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:28.228677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:28.229316 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:28.229660 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:28.229957 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 04:56:28.230306 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":384,"threads_started":1}
I20260806 04:56:28.233908 3050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907885f5a1c54d0a9707a29f836bb8b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46825 } }
I20260806 04:56:28.234431 3050 leader_election.cc:302] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 907885f5a1c54d0a9707a29f836bb8b6; no voters:
I20260806 04:56:28.235023 3050 leader_election.cc:288] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:28.235172 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:28.235176 3055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:28.236882 3050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:28.241871 3055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [term 1 LEADER]: Becoming Leader. State: Replica: 907885f5a1c54d0a9707a29f836bb8b6, State: Running, Role: LEADER
I20260806 04:56:28.242427 3055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907885f5a1c54d0a9707a29f836bb8b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46825 } }
I20260806 04:56:28.245034 3056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "907885f5a1c54d0a9707a29f836bb8b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907885f5a1c54d0a9707a29f836bb8b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46825 } } }
I20260806 04:56:28.245349 3056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:28.245781 3057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 907885f5a1c54d0a9707a29f836bb8b6. Latest consensus state: current_term: 1 leader_uuid: "907885f5a1c54d0a9707a29f836bb8b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907885f5a1c54d0a9707a29f836bb8b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46825 } } }
I20260806 04:56:28.246017 3057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:28.249060 3070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:28.256141 3070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:28.278445 3070 catalog_manager.cc:1347] Generated new cluster ID: 83944def5bf2495bb4504e92856c349d
I20260806 04:56:28.278543 3070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:28.279637 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:28.256870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:44459 (local address 127.0.0.1:46825)
0806 04:56:28.257075 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:28.257084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:28.261621 (+ 4537us) server_negotiation.cc:407] Connection header received
0806 04:56:28.261743 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:28.261748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:28.261813 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:28.261906 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:28.263638 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:28.264807 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:28.267007 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:28.267821 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:28.267926 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:28.271656 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:28.271664 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:28.271670 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:28.271714 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:28.275698 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:28.276667 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:28.276672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:28.276675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:28.276841 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:28.277422 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:28.277426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:28.277429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:28.277832 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:28.278076 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:28.279119 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0806 04:56:28.279298 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 04:56:28.288575 3070 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:28.289736 3070 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:28.301470 3070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 907885f5a1c54d0a9707a29f836bb8b6: Generated new TSK 0
I20260806 04:56:28.302093 3070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:28.308243 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 2982
I20260806 04:56:28.309270 3045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:28.388417 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 2982
W20260806 04:56:28.407507 2966 connection.cc:535] client connection to 127.0.0.1:46825 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46825): Connection reset by peer (error 104)
2026-08-06T04:56:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1822 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:28 dist-test-slave-1x4g krb5kdc[3085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:28 dist-test-slave-1x4g krb5kdc[3085](info): set up 2 sockets
Aug 06 04:56:28 dist-test-slave-1x4g krb5kdc[3085](info): commencing operation
krb5kdc: starting...
W20260806 04:56:29.400696 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:29 dist-test-slave-1x4g krb5kdc[3085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:56:29.643401 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:29.943023 3098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:29.943308 3098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:29.943461 3098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:29.949508 3098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:29.949641 3098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:29.949751 3098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:29.949870 3098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:29.949956 3098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:29.958299 3098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:29.959376 3098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:29.960620 3098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:29.972312 3103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:29.972271 3106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:29.972347 3104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:29.973284 3098 server_base.cc:1034] running on GCE node
I20260806 04:56:29.973883 3098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:29.974982 3098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:29.976161 3098 hybrid_clock.cc:648] HybridClock initialized: now 1785992189976115 us; error 41 us; skew 500 ppm
I20260806 04:56:29.976466 3098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:29 dist-test-slave-1x4g krb5kdc[3085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:29.979439 3098 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:56:29.980873 3098 webserver.cc:460] Webserver started at http://127.0.0.1:38481/ using document root <none> and password file <none>
I20260806 04:56:29.981438 3098 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:29.981529 3098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:29.981739 3098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:29.984208 3098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "f61e6da65ca5409885a404ed1f2dafb1"
format_stamp: "Formatted at 2026-08-06 04:56:29 on dist-test-slave-1x4g"
server_key: "80f59c53666b0fad7f04e723c6d14bb9"
server_key_iv: "4674c1da94776d4efcbc1e66a67322e6"
server_key_version: "encryptionkey@0"
I20260806 04:56:29.984884 3098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "f61e6da65ca5409885a404ed1f2dafb1"
format_stamp: "Formatted at 2026-08-06 04:56:29 on dist-test-slave-1x4g"
server_key: "80f59c53666b0fad7f04e723c6d14bb9"
server_key_iv: "4674c1da94776d4efcbc1e66a67322e6"
server_key_version: "encryptionkey@0"
I20260806 04:56:29.989184 3098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:56:29.992344 3113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:29.993561 3098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:56:29.993722 3098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "f61e6da65ca5409885a404ed1f2dafb1"
format_stamp: "Formatted at 2026-08-06 04:56:29 on dist-test-slave-1x4g"
server_key: "80f59c53666b0fad7f04e723c6d14bb9"
server_key_iv: "4674c1da94776d4efcbc1e66a67322e6"
server_key_version: "encryptionkey@0"
I20260806 04:56:29.993891 3098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:30.016033 3098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:30.020622 3098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:30.021008 3098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:30.039539 3098 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32885
I20260806 04:56:30.039572 3165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32885 every 8 connection(s)
I20260806 04:56:30.041011 3098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:30.044724 3166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:30.046969 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 3098
I20260806 04:56:30.047156 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:30.047657 13906 external_mini_cluster.cc:1428] Setting key aadfb6794c412587552ecd09ecfb6193
I20260806 04:56:30.056360 3166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1: Bootstrap starting.
I20260806 04:56:30.060166 3166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:30.061533 3166 log.cc:824] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:30.061652 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:30.049994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:49735 (local address 127.0.0.1:32885)
0806 04:56:30.050791 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:30.050802 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:30.050844 (+ 42us) server_negotiation.cc:407] Connection header received
0806 04:56:30.052488 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:30.052510 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:30.052964 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:30.053325 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:30.053850 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:30.055151 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:30.056558 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:30.057611 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:30.057802 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:30.058917 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:30.058927 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:30.058942 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:30.059635 (+ 693us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:30.059754 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:30.059787 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:30.059969 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:30.060142 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 04:56:30.060318 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":355,"threads_started":1}
I20260806 04:56:30.064416 3166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1: No bootstrap required, opened a new log
I20260806 04:56:30.065390 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:30.072427 3166 raft_consensus.cc:348] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61e6da65ca5409885a404ed1f2dafb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } }
I20260806 04:56:30.072902 3166 raft_consensus.cc:374] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:30.072999 3166 raft_consensus.cc:729] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61e6da65ca5409885a404ed1f2dafb1, State: Initialized, Role: FOLLOWER
I20260806 04:56:30.073577 3166 consensus_queue.cc:260] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61e6da65ca5409885a404ed1f2dafb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } }
I20260806 04:56:30.073824 3166 raft_consensus.cc:388] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:30.073966 3166 raft_consensus.cc:482] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:30.074131 3166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:30.075933 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:30.069438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:52903 (local address 127.0.0.1:32885)
0806 04:56:30.069611 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:30.069616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:30.070174 (+ 558us) server_negotiation.cc:407] Connection header received
0806 04:56:30.070338 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:30.070341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:30.070404 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:30.070480 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:30.071108 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:30.071800 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:30.073531 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:30.074020 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:30.074144 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:30.075173 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:30.075177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:30.075180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:30.075211 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:30.075264 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:30.075281 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:30.075360 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:30.075567 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 04:56:30.075604 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:56:30.076942 3166 raft_consensus.cc:504] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61e6da65ca5409885a404ed1f2dafb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } }
I20260806 04:56:30.077416 3166 leader_election.cc:302] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f61e6da65ca5409885a404ed1f2dafb1; no voters:
I20260806 04:56:30.077962 3166 leader_election.cc:288] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:30.078096 3176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:30.078801 3176 raft_consensus.cc:686] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [term 1 LEADER]: Becoming Leader. State: Replica: f61e6da65ca5409885a404ed1f2dafb1, State: Running, Role: LEADER
I20260806 04:56:30.079353 3176 consensus_queue.cc:237] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61e6da65ca5409885a404ed1f2dafb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } }
I20260806 04:56:30.080075 3166 sys_catalog.cc:564] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:30.082022 3178 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f61e6da65ca5409885a404ed1f2dafb1. Latest consensus state: current_term: 1 leader_uuid: "f61e6da65ca5409885a404ed1f2dafb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61e6da65ca5409885a404ed1f2dafb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } } }
I20260806 04:56:30.082319 3178 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:30.082841 3177 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f61e6da65ca5409885a404ed1f2dafb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61e6da65ca5409885a404ed1f2dafb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } } }
I20260806 04:56:30.083125 3177 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:30.083281 3181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:30.089383 3181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:30.099534 3181 catalog_manager.cc:1347] Generated new cluster ID: c08aad8498cf4728a8c3e2a4ad88e704
I20260806 04:56:30.099646 3181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:30.107604 3181 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:30.108740 3181 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:30.117341 3181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f61e6da65ca5409885a404ed1f2dafb1: Generated new TSK 0
I20260806 04:56:30.118076 3181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:30.127846 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 3098
W20260806 04:56:30.128921 3161 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.163:49735
W20260806 04:56:30.129606 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 3098: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:56:30.129753 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 3098
W20260806 04:56:30.148352 3083 connection.cc:535] client connection to 127.0.0.1:32885 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32885): Connection reset by peer (error 104)
2026-08-06T04:56:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1740 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:30 dist-test-slave-1x4g krb5kdc[3199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:30 dist-test-slave-1x4g krb5kdc[3199](info): set up 2 sockets
Aug 06 04:56:30 dist-test-slave-1x4g krb5kdc[3199](info): commencing operation
krb5kdc: starting...
W20260806 04:56:31.136294 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:31 dist-test-slave-1x4g krb5kdc[3199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:56:31.372062 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:31.681085 3212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:31.681348 3212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:31.681434 3212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:31.687551 3212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:31.687637 3212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:31.687677 3212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:31.687721 3212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:31.687752 3212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:31.696439 3212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:31.697644 3212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:31.698796 3212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:31.709785 3217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:31.709718 3218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:31.709820 3221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:31.710495 3212 server_base.cc:1034] running on GCE node
I20260806 04:56:31.711278 3212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:31.712563 3212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:31.713780 3212 hybrid_clock.cc:648] HybridClock initialized: now 1785992191713722 us; error 69 us; skew 500 ppm
I20260806 04:56:31.714097 3212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:31 dist-test-slave-1x4g krb5kdc[3199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:31.721633 3212 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:56:31.723006 3212 webserver.cc:460] Webserver started at http://127.0.0.1:45759/ using document root <none> and password file <none>
I20260806 04:56:31.723624 3212 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:31.723753 3212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:31.724041 3212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:31.726765 3212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "c6b87689c5944ff5882241a9b5d432d6"
format_stamp: "Formatted at 2026-08-06 04:56:31 on dist-test-slave-1x4g"
server_key: "5fcae59b8af0a60df0a5b25656802abc"
server_key_iv: "94b950bbf254c9e26d8d12794756cf6b"
server_key_version: "encryptionkey@0"
I20260806 04:56:31.727504 3212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "c6b87689c5944ff5882241a9b5d432d6"
format_stamp: "Formatted at 2026-08-06 04:56:31 on dist-test-slave-1x4g"
server_key: "5fcae59b8af0a60df0a5b25656802abc"
server_key_iv: "94b950bbf254c9e26d8d12794756cf6b"
server_key_version: "encryptionkey@0"
I20260806 04:56:31.732344 3212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260806 04:56:31.735325 3228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:31.736451 3212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:56:31.736626 3212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "c6b87689c5944ff5882241a9b5d432d6"
format_stamp: "Formatted at 2026-08-06 04:56:31 on dist-test-slave-1x4g"
server_key: "5fcae59b8af0a60df0a5b25656802abc"
server_key_iv: "94b950bbf254c9e26d8d12794756cf6b"
server_key_version: "encryptionkey@0"
I20260806 04:56:31.736853 3212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:31.742067 3212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:31.745859 3212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:31.746130 3212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:31.763515 3212 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40243
I20260806 04:56:31.763561 3280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40243 every 8 connection(s)
I20260806 04:56:31.764770 3212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:31.765765 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 3212
I20260806 04:56:31.765930 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:31.766396 13906 external_mini_cluster.cc:1428] Setting key 75e0cfb1a0da8c27da8f987c7caa0096
I20260806 04:56:31.768954 3281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:31.773626 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:31.768013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:41133 (local address 127.0.0.1:40243)
0806 04:56:31.768743 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:31.768753 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:31.768837 (+ 84us) server_negotiation.cc:407] Connection header received
0806 04:56:31.770500 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:31.770521 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:31.770834 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:31.771161 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:31.772127 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:31.772132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:31.772144 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:31.772157 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:31.772224 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:31.772246 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:31.772420 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:31.772511 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 04:56:31.772705 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":333,"threads_started":1}
I20260806 04:56:31.776875 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:31.780652 3281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6: Bootstrap starting.
I20260806 04:56:31.784228 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:31.782054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:52257 (local address 127.0.0.1:40243)
0806 04:56:31.782187 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:31.782191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:31.782574 (+ 383us) server_negotiation.cc:407] Connection header received
0806 04:56:31.782734 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:31.782737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:31.782775 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:31.782847 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:31.783680 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:31.783683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:31.783686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:31.783698 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:31.783730 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:31.783746 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:31.783821 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:31.783927 (+ 106us) server_negotiation.cc:299] Negotiation successful
0806 04:56:31.783967 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 04:56:31.784404 3281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:31.785483 3281 log.cc:824] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:31.787876 3281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6: No bootstrap required, opened a new log
I20260806 04:56:31.794483 3281 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b87689c5944ff5882241a9b5d432d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40243 } }
I20260806 04:56:31.794847 3281 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:31.794950 3281 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b87689c5944ff5882241a9b5d432d6, State: Initialized, Role: FOLLOWER
I20260806 04:56:31.795444 3281 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b87689c5944ff5882241a9b5d432d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40243 } }
I20260806 04:56:31.795665 3281 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:31.795799 3281 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:31.795955 3281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:31.797986 3281 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b87689c5944ff5882241a9b5d432d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40243 } }
I20260806 04:56:31.798496 3281 leader_election.cc:302] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6b87689c5944ff5882241a9b5d432d6; no voters:
I20260806 04:56:31.799052 3281 leader_election.cc:288] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:31.799170 3291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:31.799803 3291 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [term 1 LEADER]: Becoming Leader. State: Replica: c6b87689c5944ff5882241a9b5d432d6, State: Running, Role: LEADER
I20260806 04:56:31.800372 3291 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b87689c5944ff5882241a9b5d432d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40243 } }
I20260806 04:56:31.801067 3281 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:31.802995 3292 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6b87689c5944ff5882241a9b5d432d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b87689c5944ff5882241a9b5d432d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40243 } } }
I20260806 04:56:31.803421 3292 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:31.802969 3293 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6b87689c5944ff5882241a9b5d432d6. Latest consensus state: current_term: 1 leader_uuid: "c6b87689c5944ff5882241a9b5d432d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b87689c5944ff5882241a9b5d432d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40243 } } }
I20260806 04:56:31.803668 3293 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:31.805060 3297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:31.810854 3297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:31.820417 3297 catalog_manager.cc:1347] Generated new cluster ID: 9dc2584f5c2a49d39308270c6977ec69
I20260806 04:56:31.820520 3297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:31.828480 3297 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:31.828855 3297 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:31.835421 3297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6b87689c5944ff5882241a9b5d432d6: Generated new TSK 0
I20260806 04:56:31.836061 3297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:31.846139 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 3212
W20260806 04:56:31.847174 3275 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.163:41133
W20260806 04:56:31.847827 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 3212: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:56:31.847954 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 3212
2026-08-06T04:56:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1717 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:31 dist-test-slave-1x4g krb5kdc[3313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:31 dist-test-slave-1x4g krb5kdc[3313](info): set up 2 sockets
Aug 06 04:56:31 dist-test-slave-1x4g krb5kdc[3313](info): commencing operation
krb5kdc: starting...
W20260806 04:56:32.870787 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:33 dist-test-slave-1x4g krb5kdc[3313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:56:33.113368 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33529
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:33.416997 3326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:33.417268 3326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:33.417423 3326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:33.423738 3326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:33.423830 3326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:33.423872 3326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:33.423944 3326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:33.424028 3326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:33.432489 3326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33529
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:33.433682 3326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:33.434868 3326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:33.445972 3331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:33.445976 3334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:33.445935 3332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:33.446601 3326 server_base.cc:1034] running on GCE node
I20260806 04:56:33.447312 3326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:33.448494 3326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:33.449703 3326 hybrid_clock.cc:648] HybridClock initialized: now 1785992193449661 us; error 37 us; skew 500 ppm
I20260806 04:56:33.450042 3326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:33 dist-test-slave-1x4g krb5kdc[3313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:33.453037 3326 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:56:33.454339 3326 webserver.cc:460] Webserver started at http://127.0.0.1:40285/ using document root <none> and password file <none>
I20260806 04:56:33.454946 3326 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:33.455063 3326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:33.455343 3326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:33.458289 3326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "807c2df0241a4f3d8c389f8b44782977"
format_stamp: "Formatted at 2026-08-06 04:56:33 on dist-test-slave-1x4g"
server_key: "faae66a40dd0b04100be809cdf90baf8"
server_key_iv: "e1f2eae60d1f6afa996a3300b747445b"
server_key_version: "encryptionkey@0"
I20260806 04:56:33.458951 3326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "807c2df0241a4f3d8c389f8b44782977"
format_stamp: "Formatted at 2026-08-06 04:56:33 on dist-test-slave-1x4g"
server_key: "faae66a40dd0b04100be809cdf90baf8"
server_key_iv: "e1f2eae60d1f6afa996a3300b747445b"
server_key_version: "encryptionkey@0"
I20260806 04:56:33.463654 3326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:56:33.466711 3341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:33.467808 3326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 04:56:33.467973 3326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "807c2df0241a4f3d8c389f8b44782977"
format_stamp: "Formatted at 2026-08-06 04:56:33 on dist-test-slave-1x4g"
server_key: "faae66a40dd0b04100be809cdf90baf8"
server_key_iv: "e1f2eae60d1f6afa996a3300b747445b"
server_key_version: "encryptionkey@0"
I20260806 04:56:33.468134 3326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:33.480479 3326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:33.484000 3326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:33.484279 3326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:33.501055 3326 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33529
I20260806 04:56:33.501085 3392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33529 every 8 connection(s)
I20260806 04:56:33.502316 3326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:33.505579 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 3326
I20260806 04:56:33.505766 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:33.506110 3393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:33.506289 13906 external_mini_cluster.cc:1428] Setting key d0844c8e27fa9a6b2a94aab6f5ba90d2
I20260806 04:56:33.520118 3393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977: Bootstrap starting.
Aug 06 04:56:33 dist-test-slave-1x4g krb5kdc[3313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 04:56:33.523603 3393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:33.524715 3393 log.cc:824] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:33.527326 3393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977: No bootstrap required, opened a new log
I20260806 04:56:33.528493 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:33.507962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:50691 (local address 127.0.0.1:33529)
0806 04:56:33.508692 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:33.508703 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:33.508744 (+ 41us) server_negotiation.cc:407] Connection header received
0806 04:56:33.510612 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:33.510633 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:33.511034 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:33.511376 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:33.512965 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:33.514273 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:33.515961 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:33.516608 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:33.516756 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:33.521818 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:33.521825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:33.521837 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:33.521874 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:33.524687 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:33.525500 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:33.525506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:33.525516 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:33.525670 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:33.526126 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:33.526130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:33.526131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:33.526616 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:33.526910 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:33.527361 (+ 451us) server_negotiation.cc:299] Negotiation successful
0806 04:56:33.527648 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":341,"threads_started":1}
I20260806 04:56:33.532065 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:33.534650 3393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807c2df0241a4f3d8c389f8b44782977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33529 } }
I20260806 04:56:33.535048 3393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:33.535148 3393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 807c2df0241a4f3d8c389f8b44782977, State: Initialized, Role: FOLLOWER
I20260806 04:56:33.535640 3393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807c2df0241a4f3d8c389f8b44782977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33529 } }
I20260806 04:56:33.535881 3393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:33.535996 3393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:33.536108 3393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:33.538424 3393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807c2df0241a4f3d8c389f8b44782977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33529 } }
I20260806 04:56:33.538877 3393 leader_election.cc:302] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 807c2df0241a4f3d8c389f8b44782977; no voters:
I20260806 04:56:33.539466 3393 leader_election.cc:288] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:33.539535 3403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:33.541548 3393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:33.546344 3403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [term 1 LEADER]: Becoming Leader. State: Replica: 807c2df0241a4f3d8c389f8b44782977, State: Running, Role: LEADER
I20260806 04:56:33.546818 3403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807c2df0241a4f3d8c389f8b44782977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33529 } }
I20260806 04:56:33.549734 3405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 807c2df0241a4f3d8c389f8b44782977. Latest consensus state: current_term: 1 leader_uuid: "807c2df0241a4f3d8c389f8b44782977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807c2df0241a4f3d8c389f8b44782977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33529 } } }
I20260806 04:56:33.550127 3405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:33.553329 3404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "807c2df0241a4f3d8c389f8b44782977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807c2df0241a4f3d8c389f8b44782977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33529 } } }
I20260806 04:56:33.554076 3404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:33.555236 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:33.536082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:41783 (local address 127.0.0.1:33529)
0806 04:56:33.536258 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:33.536263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:33.537359 (+ 1096us) server_negotiation.cc:407] Connection header received
0806 04:56:33.537480 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:33.537485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:33.537558 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:33.537661 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:33.538924 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:33.540074 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:33.541860 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:33.542502 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:33.542624 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:33.546164 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:33.546173 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:33.546183 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:33.546230 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:33.551102 (+ 4872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:33.552224 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:33.552232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:33.552237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:33.552391 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:33.553028 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:33.553035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:33.553040 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:33.553480 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:33.553798 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:33.554550 (+ 752us) server_negotiation.cc:299] Negotiation successful
0806 04:56:33.554782 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:56:33.562286 3409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:33.566797 3409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:33.575089 3409 catalog_manager.cc:1347] Generated new cluster ID: 250f24762b614f9fb915c27bffa3d4e7
I20260806 04:56:33.575192 3409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:33.586736 3409 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:33.588374 3409 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:33.596271 3409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 807c2df0241a4f3d8c389f8b44782977: Generated new TSK 0
I20260806 04:56:33.597302 3409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:33.608474 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 3326
I20260806 04:56:33.609565 3388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:33.693169 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 3326
W20260806 04:56:33.711374 3310 connection.cc:535] client connection to 127.0.0.1:33529 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33529): Connection reset by peer (error 104)
2026-08-06T04:56:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1847 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:33 dist-test-slave-1x4g krb5kdc[3426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:33 dist-test-slave-1x4g krb5kdc[3426](info): set up 2 sockets
Aug 06 04:56:33 dist-test-slave-1x4g krb5kdc[3426](info): commencing operation
krb5kdc: starting...
W20260806 04:56:34.717425 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:34 dist-test-slave-1x4g krb5kdc[3426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:56:34.943722 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:35.248644 3439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:35.248956 3439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:35.249104 3439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:35.255328 3439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:35.255457 3439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:35.255555 3439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:35.255664 3439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:35.255745 3439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:35.264226 3439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37127
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:35.265384 3439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:35.266542 3439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:35.277192 3445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:35.277228 3444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:35.277281 3447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:35.278623 3439 server_base.cc:1034] running on GCE node
I20260806 04:56:35.279251 3439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:35.280405 3439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:35.281626 3439 hybrid_clock.cc:648] HybridClock initialized: now 1785992195281594 us; error 45 us; skew 500 ppm
I20260806 04:56:35.281946 3439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:35 dist-test-slave-1x4g krb5kdc[3426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:35.284883 3439 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:56:35.286166 3439 webserver.cc:460] Webserver started at http://127.0.0.1:44369/ using document root <none> and password file <none>
I20260806 04:56:35.286754 3439 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:35.286846 3439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:35.287067 3439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:35.289788 3439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "68531946638c43e39bb97367e2fa271a"
format_stamp: "Formatted at 2026-08-06 04:56:35 on dist-test-slave-1x4g"
server_key: "99c22362d2beb7019701dcd50297540a"
server_key_iv: "ad5229f5e3366c3fc84bcd3c7b2c549a"
server_key_version: "encryptionkey@0"
I20260806 04:56:35.290488 3439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "68531946638c43e39bb97367e2fa271a"
format_stamp: "Formatted at 2026-08-06 04:56:35 on dist-test-slave-1x4g"
server_key: "99c22362d2beb7019701dcd50297540a"
server_key_iv: "ad5229f5e3366c3fc84bcd3c7b2c549a"
server_key_version: "encryptionkey@0"
I20260806 04:56:35.295276 3439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:56:35.298525 3454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:35.299690 3439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:56:35.299850 3439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "68531946638c43e39bb97367e2fa271a"
format_stamp: "Formatted at 2026-08-06 04:56:35 on dist-test-slave-1x4g"
server_key: "99c22362d2beb7019701dcd50297540a"
server_key_iv: "ad5229f5e3366c3fc84bcd3c7b2c549a"
server_key_version: "encryptionkey@0"
I20260806 04:56:35.300014 3439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:35.322794 3439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:35.326431 3439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:35.326676 3439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:35.343986 3439 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37127
I20260806 04:56:35.344055 3507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37127 every 8 connection(s)
I20260806 04:56:35.345378 3439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:35.346738 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 3439
I20260806 04:56:35.346928 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:35.347471 13906 external_mini_cluster.cc:1428] Setting key b3e80948f8949d2bbd2bf6ff28bd7e20
I20260806 04:56:35.349560 3508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:35.359736 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:35.349171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:39767 (local address 127.0.0.1:37127)
0806 04:56:35.349912 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:35.349923 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:35.349960 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:56:35.351619 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:35.351638 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:35.351985 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:35.352283 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:35.352911 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:35.354077 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:35.355703 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:35.356264 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:35.356375 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:35.357692 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:35.357699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:35.357714 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:35.358220 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:35.358287 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:35.358314 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:35.358424 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:35.358740 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 04:56:35.358912 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":351,"threads_started":1}
I20260806 04:56:35.361186 3508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a: Bootstrap starting.
I20260806 04:56:35.363308 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:35.364157 3508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:35.365108 3508 log.cc:824] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:35.367691 3508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a: No bootstrap required, opened a new log
I20260806 04:56:35.373952 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:35.367382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:37565 (local address 127.0.0.1:37127)
0806 04:56:35.367566 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:35.367571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:35.368281 (+ 710us) server_negotiation.cc:407] Connection header received
0806 04:56:35.368426 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:35.368429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:35.368476 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:35.368553 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:35.369113 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:35.369815 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:35.371666 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:35.372053 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:35.372132 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:35.373245 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:35.373249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:35.373252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:35.373265 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:35.373310 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:35.373323 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:35.373401 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:35.373605 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 04:56:35.373641 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 04:56:35.376367 3508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68531946638c43e39bb97367e2fa271a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } }
I20260806 04:56:35.376716 3508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:35.376842 3508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68531946638c43e39bb97367e2fa271a, State: Initialized, Role: FOLLOWER
I20260806 04:56:35.377372 3508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68531946638c43e39bb97367e2fa271a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } }
I20260806 04:56:35.377612 3508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:35.377755 3508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:35.377930 3508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:35.379917 3508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68531946638c43e39bb97367e2fa271a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } }
I20260806 04:56:35.380429 3508 leader_election.cc:302] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68531946638c43e39bb97367e2fa271a; no voters:
I20260806 04:56:35.381093 3508 leader_election.cc:288] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:35.381191 3518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:35.381906 3518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [term 1 LEADER]: Becoming Leader. State: Replica: 68531946638c43e39bb97367e2fa271a, State: Running, Role: LEADER
I20260806 04:56:35.382354 3518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68531946638c43e39bb97367e2fa271a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } }
I20260806 04:56:35.383056 3508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:35.385196 3519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68531946638c43e39bb97367e2fa271a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68531946638c43e39bb97367e2fa271a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } } }
I20260806 04:56:35.385639 3519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:35.386116 3520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68531946638c43e39bb97367e2fa271a. Latest consensus state: current_term: 1 leader_uuid: "68531946638c43e39bb97367e2fa271a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68531946638c43e39bb97367e2fa271a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } } }
I20260806 04:56:35.386420 3520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:35.387015 3526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:35.390914 3526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:35.399600 3526 catalog_manager.cc:1347] Generated new cluster ID: e5d5421f17d14d2fbf76c11d2d4fdc30
I20260806 04:56:35.399694 3526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:35.415928 3526 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:35.417160 3526 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:35.425827 3526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68531946638c43e39bb97367e2fa271a: Generated new TSK 0
I20260806 04:56:35.426597 3526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:35.435101 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 3439
W20260806 04:56:35.436094 3503 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.163:39767
W20260806 04:56:35.436866 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 3439: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:56:35.437028 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 3439
W20260806 04:56:35.455919 3424 connection.cc:535] client connection to 127.0.0.1:37127 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37127): Connection reset by peer (error 104)
2026-08-06T04:56:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1742 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:35 dist-test-slave-1x4g krb5kdc[3539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:35 dist-test-slave-1x4g krb5kdc[3539](info): set up 2 sockets
Aug 06 04:56:35 dist-test-slave-1x4g krb5kdc[3539](info): commencing operation
krb5kdc: starting...
W20260806 04:56:36.445923 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:36 dist-test-slave-1x4g krb5kdc[3539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:56:36.679432 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35651
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:36.983654 3553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:36.983891 3553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:36.983973 3553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:36.991657 3553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:36.991745 3553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:36.991792 3553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:36.991845 3553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:36.991882 3553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:37.000617 3553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35651
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:37.001829 3553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:37.002959 3553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:37.014479 3560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:37.014520 3562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:37.014567 3559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:37.015363 3553 server_base.cc:1034] running on GCE node
I20260806 04:56:37.016040 3553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:37.017359 3553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:37.018649 3553 hybrid_clock.cc:648] HybridClock initialized: now 1785992197018586 us; error 49 us; skew 500 ppm
I20260806 04:56:37.018960 3553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:37 dist-test-slave-1x4g krb5kdc[3539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:37.025913 3553 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:56:37.027202 3553 webserver.cc:460] Webserver started at http://127.0.0.1:39971/ using document root <none> and password file <none>
I20260806 04:56:37.027822 3553 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:37.027940 3553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:37.028227 3553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:37.031078 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "a91b94e16ac44b8d9d910e7d75255a88"
format_stamp: "Formatted at 2026-08-06 04:56:37 on dist-test-slave-1x4g"
server_key: "e35f38ba57332156fa4acecbced71ee8"
server_key_iv: "e4d0a307df2cd0a82758669ab0dd3ee9"
server_key_version: "encryptionkey@0"
I20260806 04:56:37.031810 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "a91b94e16ac44b8d9d910e7d75255a88"
format_stamp: "Formatted at 2026-08-06 04:56:37 on dist-test-slave-1x4g"
server_key: "e35f38ba57332156fa4acecbced71ee8"
server_key_iv: "e4d0a307df2cd0a82758669ab0dd3ee9"
server_key_version: "encryptionkey@0"
I20260806 04:56:37.036553 3553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260806 04:56:37.039999 3569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:37.041386 3553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:56:37.041555 3553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "a91b94e16ac44b8d9d910e7d75255a88"
format_stamp: "Formatted at 2026-08-06 04:56:37 on dist-test-slave-1x4g"
server_key: "e35f38ba57332156fa4acecbced71ee8"
server_key_iv: "e4d0a307df2cd0a82758669ab0dd3ee9"
server_key_version: "encryptionkey@0"
I20260806 04:56:37.041771 3553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:37.047144 3553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:37.051009 3553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:37.051292 3553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:37.068397 3553 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35651
I20260806 04:56:37.068476 3621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35651 every 8 connection(s)
I20260806 04:56:37.069679 3553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:37.071339 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 3553
I20260806 04:56:37.071563 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:37.071955 13906 external_mini_cluster.cc:1428] Setting key c97512907d190b7cd060e4e1e4fd34c2
I20260806 04:56:37.073681 3622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:37.078801 3625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:37.073539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:37953 (local address 127.0.0.1:35651)
0806 04:56:37.074277 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:37.074288 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:37.074326 (+ 38us) server_negotiation.cc:407] Connection header received
0806 04:56:37.075715 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:37.075734 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:37.076034 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:37.076317 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:37.077338 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:37.077344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:37.077355 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:37.077368 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:37.077444 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:37.077466 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:37.077624 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:37.077740 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 04:56:37.077969 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":348,"threads_started":1}
I20260806 04:56:37.082167 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:37.084415 3622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88: Bootstrap starting.
I20260806 04:56:37.087983 3622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:37.088534 3625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:37.086147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:58397 (local address 127.0.0.1:35651)
0806 04:56:37.086288 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:37.086292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:37.086626 (+ 334us) server_negotiation.cc:407] Connection header received
0806 04:56:37.086778 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:37.086781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:37.086832 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:37.086896 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:37.087819 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:37.087823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:37.087826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:37.087840 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:37.087878 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:37.087892 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:37.087987 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:37.088154 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 04:56:37.088211 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 04:56:37.089262 3622 log.cc:824] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:37.091764 3622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88: No bootstrap required, opened a new log
I20260806 04:56:37.098384 3622 raft_consensus.cc:348] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91b94e16ac44b8d9d910e7d75255a88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35651 } }
I20260806 04:56:37.098730 3622 raft_consensus.cc:374] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:37.098832 3622 raft_consensus.cc:729] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b94e16ac44b8d9d910e7d75255a88, State: Initialized, Role: FOLLOWER
I20260806 04:56:37.099397 3622 consensus_queue.cc:260] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91b94e16ac44b8d9d910e7d75255a88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35651 } }
I20260806 04:56:37.099612 3622 raft_consensus.cc:388] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:37.099758 3622 raft_consensus.cc:482] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:37.099917 3622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:37.102059 3622 raft_consensus.cc:504] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91b94e16ac44b8d9d910e7d75255a88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35651 } }
I20260806 04:56:37.102517 3622 leader_election.cc:302] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a91b94e16ac44b8d9d910e7d75255a88; no voters:
I20260806 04:56:37.103048 3622 leader_election.cc:288] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:37.103188 3632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:37.104225 3632 raft_consensus.cc:686] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [term 1 LEADER]: Becoming Leader. State: Replica: a91b94e16ac44b8d9d910e7d75255a88, State: Running, Role: LEADER
I20260806 04:56:37.104652 3632 consensus_queue.cc:237] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91b94e16ac44b8d9d910e7d75255a88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35651 } }
I20260806 04:56:37.105033 3622 sys_catalog.cc:564] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:37.107115 3634 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a91b94e16ac44b8d9d910e7d75255a88. Latest consensus state: current_term: 1 leader_uuid: "a91b94e16ac44b8d9d910e7d75255a88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91b94e16ac44b8d9d910e7d75255a88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35651 } } }
I20260806 04:56:37.107477 3634 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:37.108024 3633 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a91b94e16ac44b8d9d910e7d75255a88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91b94e16ac44b8d9d910e7d75255a88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35651 } } }
I20260806 04:56:37.108351 3633 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:37.108451 3641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:37.115485 3641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:37.124374 3641 catalog_manager.cc:1347] Generated new cluster ID: df09ff23a8144f02ba0f9244d4b28712
I20260806 04:56:37.124480 3641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:37.154618 3641 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:37.155020 3641 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:37.161974 3641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a91b94e16ac44b8d9d910e7d75255a88: Generated new TSK 0
I20260806 04:56:37.162724 3641 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:37.168223 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 3553
W20260806 04:56:37.169342 3617 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.163:37953
W20260806 04:56:37.169951 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 3553: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:56:37.170084 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 3553
2026-08-06T04:56:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1732 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:37 dist-test-slave-1x4g krb5kdc[3654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:37 dist-test-slave-1x4g krb5kdc[3654](info): set up 2 sockets
Aug 06 04:56:37 dist-test-slave-1x4g krb5kdc[3654](info): commencing operation
krb5kdc: starting...
W20260806 04:56:38.188704 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:38 dist-test-slave-1x4g krb5kdc[3654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:56:38.421633 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:38243
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:38243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:38.728018 3667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:38.728288 3667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:38.728382 3667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:38.734735 3667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:38.734826 3667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:38.734885 3667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:38.734938 3667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:38.734983 3667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:38.743490 3667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:38243
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:38243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:38.744628 3667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:38.745872 3667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:38.757313 3673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:38.757603 3675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:38.757293 3672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:38.758395 3667 server_base.cc:1034] running on GCE node
I20260806 04:56:38.759128 3667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:38.760327 3667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:38.761559 3667 hybrid_clock.cc:648] HybridClock initialized: now 1785992198761514 us; error 42 us; skew 500 ppm
I20260806 04:56:38.761905 3667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:38 dist-test-slave-1x4g krb5kdc[3654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992198, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:38.764706 3667 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:56:38.765928 3667 webserver.cc:460] Webserver started at http://127.13.148.190:44409/ using document root <none> and password file <none>
I20260806 04:56:38.766585 3667 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:38.766719 3667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:38.767000 3667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:38.769699 3667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "3f49d9a8ad4a4be6896db1813e798087"
format_stamp: "Formatted at 2026-08-06 04:56:38 on dist-test-slave-1x4g"
server_key: "0760449df94b68afd110de9ddffa7e0c"
server_key_iv: "f69349c829c904ee871ee72d27c84950"
server_key_version: "encryptionkey@0"
I20260806 04:56:38.770398 3667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "3f49d9a8ad4a4be6896db1813e798087"
format_stamp: "Formatted at 2026-08-06 04:56:38 on dist-test-slave-1x4g"
server_key: "0760449df94b68afd110de9ddffa7e0c"
server_key_iv: "f69349c829c904ee871ee72d27c84950"
server_key_version: "encryptionkey@0"
I20260806 04:56:38.775163 3667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:56:38.778496 3682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:38.779676 3667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:56:38.779850 3667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "3f49d9a8ad4a4be6896db1813e798087"
format_stamp: "Formatted at 2026-08-06 04:56:38 on dist-test-slave-1x4g"
server_key: "0760449df94b68afd110de9ddffa7e0c"
server_key_iv: "f69349c829c904ee871ee72d27c84950"
server_key_version: "encryptionkey@0"
I20260806 04:56:38.780030 3667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:38.803604 3667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:38.806910 3667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:38.807157 3667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:38.824527 3667 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:38243
I20260806 04:56:38.824563 3735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:38243 every 8 connection(s)
I20260806 04:56:38.825719 3667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:38.829087 3736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:38.831642 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 3667
I20260806 04:56:38.831800 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:38.832278 13906 external_mini_cluster.cc:1428] Setting key 2d4a6eb7d3614285fb3af4b7f5d05426
I20260806 04:56:38.844209 3736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087: Bootstrap starting.
Aug 06 04:56:38 dist-test-slave-1x4g krb5kdc[3654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:56:38.847620 3736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:38.848875 3736 log.cc:824] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:38.851915 3736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087: No bootstrap required, opened a new log
I20260806 04:56:38.855114 3739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:38.833906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46179 (local address 127.13.148.190:38243)
0806 04:56:38.834671 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:38.834685 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:38.834729 (+ 44us) server_negotiation.cc:407] Connection header received
0806 04:56:38.836906 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:38.836927 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:38.837303 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:38.837625 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:38.839318 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:38.840506 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:38.842331 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:38.842910 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:38.843022 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:38.847534 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:38.847547 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:38.847566 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:38.847613 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:38.851251 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:38.851991 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:38.851997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:38.852004 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:38.852183 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:38.852602 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:38.852609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:38.852612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:38.853181 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:38.853388 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:38.853869 (+ 481us) server_negotiation.cc:299] Negotiation successful
0806 04:56:38.854156 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":315,"threads_started":1}
I20260806 04:56:38.858721 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:38.860003 3736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f49d9a8ad4a4be6896db1813e798087" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38243 } }
I20260806 04:56:38.860417 3736 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:38.860509 3736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f49d9a8ad4a4be6896db1813e798087, State: Initialized, Role: FOLLOWER
I20260806 04:56:38.861161 3736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f49d9a8ad4a4be6896db1813e798087" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38243 } }
I20260806 04:56:38.861446 3736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:38.861624 3736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:38.861783 3736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:38.864559 3736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f49d9a8ad4a4be6896db1813e798087" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38243 } }
I20260806 04:56:38.865137 3736 leader_election.cc:302] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f49d9a8ad4a4be6896db1813e798087; no voters:
I20260806 04:56:38.865790 3736 leader_election.cc:288] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:38.865849 3746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:38.867254 3746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [term 1 LEADER]: Becoming Leader. State: Replica: 3f49d9a8ad4a4be6896db1813e798087, State: Running, Role: LEADER
I20260806 04:56:38.867806 3736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:38.867740 3746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f49d9a8ad4a4be6896db1813e798087" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38243 } }
I20260806 04:56:38.873971 3747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f49d9a8ad4a4be6896db1813e798087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f49d9a8ad4a4be6896db1813e798087" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38243 } } }
I20260806 04:56:38.874325 3747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:38.874773 3748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f49d9a8ad4a4be6896db1813e798087. Latest consensus state: current_term: 1 leader_uuid: "3f49d9a8ad4a4be6896db1813e798087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f49d9a8ad4a4be6896db1813e798087" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38243 } } }
I20260806 04:56:38.875030 3748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:38.883139 3739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:38.863543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49183 (local address 127.13.148.190:38243)
0806 04:56:38.863721 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:38.863727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:38.864184 (+ 457us) server_negotiation.cc:407] Connection header received
0806 04:56:38.864319 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:38.864322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:38.864369 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:38.864434 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:38.866256 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:38.867311 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:38.868893 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:38.869774 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:38.869917 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:38.873677 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:38.873686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:38.873692 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:38.873735 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:38.878061 (+ 4326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:38.879167 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:38.879175 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:38.879181 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:38.879344 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:38.880021 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:38.880028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:38.880033 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:38.880568 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:38.880938 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:38.882411 (+ 1473us) server_negotiation.cc:299] Negotiation successful
0806 04:56:38.882612 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 04:56:38.887831 3760 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:56:38.887944 3760 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:56:38.888060 3761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:38.890048 3761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:38.897667 3761 catalog_manager.cc:1347] Generated new cluster ID: 436b5689d83449e3929fa179c0a0d2d8
I20260806 04:56:38.897779 3761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:38.917520 3761 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:38.918601 3761 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:38.932581 3761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f49d9a8ad4a4be6896db1813e798087: Generated new TSK 0
I20260806 04:56:38.933485 3761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:38.947479 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 3667
I20260806 04:56:38.948794 3731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:39.034915 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 3667
W20260806 04:56:39.053699 3652 connection.cc:535] client connection to 127.13.148.190:38243 recv error: Network error: failed to read from TLS socket (remote: 127.13.148.190:38243): Connection reset by peer (error 104)
2026-08-06T04:56:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1865 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:39 dist-test-slave-1x4g krb5kdc[3768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:39 dist-test-slave-1x4g krb5kdc[3768](info): set up 2 sockets
Aug 06 04:56:39 dist-test-slave-1x4g krb5kdc[3768](info): commencing operation
krb5kdc: starting...
W20260806 04:56:40.050633 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:40 dist-test-slave-1x4g krb5kdc[3768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:56:40.297994 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:37355
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:41733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:37355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:40.602883 3781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:40.603178 3781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:40.603324 3781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:40.609653 3781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:40.609779 3781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:40.609872 3781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:40.609967 3781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:40.610039 3781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:40.618690 3781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:41733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:37355
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:37355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:40.619819 3781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:40.621088 3781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:40.631958 3786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:40.632081 3789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:40.632112 3787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:40.633435 3781 server_base.cc:1034] running on GCE node
I20260806 04:56:40.634146 3781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:40.635330 3781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:40.636555 3781 hybrid_clock.cc:648] HybridClock initialized: now 1785992200636500 us; error 56 us; skew 500 ppm
I20260806 04:56:40.636957 3781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:40 dist-test-slave-1x4g krb5kdc[3768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992200, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:40.639961 3781 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:56:40.641333 3781 webserver.cc:460] Webserver started at http://127.13.148.190:46613/ using document root <none> and password file <none>
I20260806 04:56:40.641990 3781 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:40.642135 3781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:40.642433 3781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:40.645051 3781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "d4a657b0f7ac4b809f0af790da0bedeb"
format_stamp: "Formatted at 2026-08-06 04:56:40 on dist-test-slave-1x4g"
server_key: "0f113cc66e20cc3f7c626c9447acf125"
server_key_iv: "a7cf99ae56f9ecd968af8d860bdea42d"
server_key_version: "encryptionkey@0"
I20260806 04:56:40.645728 3781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "d4a657b0f7ac4b809f0af790da0bedeb"
format_stamp: "Formatted at 2026-08-06 04:56:40 on dist-test-slave-1x4g"
server_key: "0f113cc66e20cc3f7c626c9447acf125"
server_key_iv: "a7cf99ae56f9ecd968af8d860bdea42d"
server_key_version: "encryptionkey@0"
I20260806 04:56:40.650172 3781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:56:40.653288 3796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:40.654352 3781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:56:40.654521 3781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "d4a657b0f7ac4b809f0af790da0bedeb"
format_stamp: "Formatted at 2026-08-06 04:56:40 on dist-test-slave-1x4g"
server_key: "0f113cc66e20cc3f7c626c9447acf125"
server_key_iv: "a7cf99ae56f9ecd968af8d860bdea42d"
server_key_version: "encryptionkey@0"
I20260806 04:56:40.654687 3781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:40.665724 3781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:40.669023 3781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:40.669276 3781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:40.686048 3781 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:37355
I20260806 04:56:40.686082 3848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:37355 every 8 connection(s)
I20260806 04:56:40.687163 3781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:40.689095 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 3781
I20260806 04:56:40.689272 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:40.689746 13906 external_mini_cluster.cc:1428] Setting key 253b16ec440ae615564846be6d86db0f
I20260806 04:56:40.690886 3849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:40.701689 3852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:40.691318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40117 (local address 127.13.148.190:37355)
0806 04:56:40.692056 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:40.692066 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:40.692102 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:56:40.693491 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:40.693509 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:40.693835 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:40.694125 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:40.694632 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:40.695783 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:40.697584 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:40.698222 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:40.698337 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:40.699473 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:40.699479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:40.699493 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:40.699988 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:40.700056 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:40.700084 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:40.700194 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:40.700471 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 04:56:40.700681 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":377,"threads_started":1}
I20260806 04:56:40.703120 3849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb: Bootstrap starting.
I20260806 04:56:40.705281 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:40.706267 3849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:40.707144 3849 log.cc:824] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:40.709777 3849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb: No bootstrap required, opened a new log
I20260806 04:56:40.714776 3852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:40.709249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58451 (local address 127.13.148.190:37355)
0806 04:56:40.709401 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:40.709406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:40.709819 (+ 413us) server_negotiation.cc:407] Connection header received
0806 04:56:40.710002 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:40.710006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:40.710056 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:40.710149 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:40.710657 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:40.711434 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:40.712593 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:40.713045 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:40.713159 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:40.714145 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:40.714149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:40.714152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:40.714165 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:40.714201 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:40.714214 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:40.714321 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:40.714487 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 04:56:40.714536 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:56:40.716707 3849 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a657b0f7ac4b809f0af790da0bedeb" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37355 } }
I20260806 04:56:40.717113 3849 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:40.717217 3849 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4a657b0f7ac4b809f0af790da0bedeb, State: Initialized, Role: FOLLOWER
I20260806 04:56:40.717720 3849 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a657b0f7ac4b809f0af790da0bedeb" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37355 } }
I20260806 04:56:40.717944 3849 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:40.718092 3849 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:40.718232 3849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:40.720088 3849 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a657b0f7ac4b809f0af790da0bedeb" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37355 } }
I20260806 04:56:40.720572 3849 leader_election.cc:302] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4a657b0f7ac4b809f0af790da0bedeb; no voters:
I20260806 04:56:40.721149 3849 leader_election.cc:288] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:40.721246 3859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:40.722440 3859 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [term 1 LEADER]: Becoming Leader. State: Replica: d4a657b0f7ac4b809f0af790da0bedeb, State: Running, Role: LEADER
I20260806 04:56:40.722874 3859 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a657b0f7ac4b809f0af790da0bedeb" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37355 } }
I20260806 04:56:40.723088 3849 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:40.725217 3861 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a657b0f7ac4b809f0af790da0bedeb. Latest consensus state: current_term: 1 leader_uuid: "d4a657b0f7ac4b809f0af790da0bedeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a657b0f7ac4b809f0af790da0bedeb" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37355 } } }
I20260806 04:56:40.725528 3861 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:40.725323 3860 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4a657b0f7ac4b809f0af790da0bedeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a657b0f7ac4b809f0af790da0bedeb" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 37355 } } }
I20260806 04:56:40.725703 3860 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:40.726336 3867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:40.731446 3867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:40.743237 3867 catalog_manager.cc:1347] Generated new cluster ID: 4938837cb5544aada000d3a63ac009da
I20260806 04:56:40.743340 3867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:40.749369 3867 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:40.750319 3867 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:40.758322 3867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4a657b0f7ac4b809f0af790da0bedeb: Generated new TSK 0
I20260806 04:56:40.759047 3867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:40.768674 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 3781
W20260806 04:56:40.769749 3843 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40117
W20260806 04:56:40.770530 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 3781: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:56:40.770679 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 3781
W20260806 04:56:40.788312 3766 connection.cc:535] client connection to 127.13.148.190:37355 recv error: Network error: failed to read from TLS socket (remote: 127.13.148.190:37355): Connection reset by peer (error 104)
2026-08-06T04:56:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1734 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:40 dist-test-slave-1x4g krb5kdc[3880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:40 dist-test-slave-1x4g krb5kdc[3880](info): set up 2 sockets
Aug 06 04:56:40 dist-test-slave-1x4g krb5kdc[3880](info): commencing operation
krb5kdc: starting...
W20260806 04:56:41.789306 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:41 dist-test-slave-1x4g krb5kdc[3880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:56:42.029418 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:33857
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:33857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:42.331686 3893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:42.331995 3893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:42.332142 3893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:42.338383 3893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:42.338495 3893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:42.338581 3893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:42.338665 3893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:42.338737 3893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:42.347218 3893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:33857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:33857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:42.348356 3893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:42.349668 3893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:42.360002 3899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:42.360415 3898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:42.360586 3901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:42.361253 3893 server_base.cc:1034] running on GCE node
I20260806 04:56:42.362268 3893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:42.363750 3893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:42.364928 3893 hybrid_clock.cc:648] HybridClock initialized: now 1785992202364881 us; error 42 us; skew 500 ppm
I20260806 04:56:42.365310 3893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:42 dist-test-slave-1x4g krb5kdc[3880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992202, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:42.372956 3893 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:56:42.374188 3893 webserver.cc:460] Webserver started at http://127.13.148.190:45469/ using document root <none> and password file <none>
I20260806 04:56:42.374781 3893 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:42.374912 3893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:42.375196 3893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:42.377997 3893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "39e53e19894643dfb0ecbd7dd734207f"
format_stamp: "Formatted at 2026-08-06 04:56:42 on dist-test-slave-1x4g"
server_key: "8868b4db03264d21fe8e73df263bfff0"
server_key_iv: "ad7c03dff1ca7b0d4e9139892c7c310b"
server_key_version: "encryptionkey@0"
I20260806 04:56:42.378649 3893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "39e53e19894643dfb0ecbd7dd734207f"
format_stamp: "Formatted at 2026-08-06 04:56:42 on dist-test-slave-1x4g"
server_key: "8868b4db03264d21fe8e73df263bfff0"
server_key_iv: "ad7c03dff1ca7b0d4e9139892c7c310b"
server_key_version: "encryptionkey@0"
I20260806 04:56:42.383260 3893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:56:42.386225 3908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:42.387329 3893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:56:42.387480 3893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "39e53e19894643dfb0ecbd7dd734207f"
format_stamp: "Formatted at 2026-08-06 04:56:42 on dist-test-slave-1x4g"
server_key: "8868b4db03264d21fe8e73df263bfff0"
server_key_iv: "ad7c03dff1ca7b0d4e9139892c7c310b"
server_key_version: "encryptionkey@0"
I20260806 04:56:42.387742 3893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:42.393086 3893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:42.396476 3893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:42.396883 3893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:42.414335 3893 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:33857
I20260806 04:56:42.414364 3959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:33857 every 8 connection(s)
I20260806 04:56:42.415549 3893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:42.418891 3960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:42.422231 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 3893
I20260806 04:56:42.422401 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:42.422895 13906 external_mini_cluster.cc:1428] Setting key a2429ef1290c670bd4a459f50c11d5da
I20260806 04:56:42.429647 3963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:42.424510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47575 (local address 127.13.148.190:33857)
0806 04:56:42.425224 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:42.425235 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:42.425271 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:56:42.426779 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:42.426799 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:42.427099 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:42.427393 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:42.428291 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:42.428299 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:42.428311 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:42.428324 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:42.428380 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:42.428398 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:42.428554 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:42.428641 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 04:56:42.428861 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":360,"threads_started":1}
I20260806 04:56:42.430321 3960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f: Bootstrap starting.
I20260806 04:56:42.432950 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:42.433384 3960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:42.434288 3960 log.cc:824] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:42.436945 3960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f: No bootstrap required, opened a new log
I20260806 04:56:42.439850 3963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:42.437436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56955 (local address 127.13.148.190:33857)
0806 04:56:42.437654 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:42.437662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:42.438042 (+ 380us) server_negotiation.cc:407] Connection header received
0806 04:56:42.438185 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:42.438188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:42.438228 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:42.438298 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:42.439216 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:42.439220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:42.439223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:42.439236 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:42.439273 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:42.439286 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:42.439372 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:42.439501 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 04:56:42.439546 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 04:56:42.445024 3960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e53e19894643dfb0ecbd7dd734207f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33857 } }
I20260806 04:56:42.445350 3960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:42.445449 3960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e53e19894643dfb0ecbd7dd734207f, State: Initialized, Role: FOLLOWER
I20260806 04:56:42.445962 3960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e53e19894643dfb0ecbd7dd734207f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33857 } }
I20260806 04:56:42.446257 3960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:42.446365 3960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:42.446470 3960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:42.448341 3960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e53e19894643dfb0ecbd7dd734207f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33857 } }
I20260806 04:56:42.448725 3960 leader_election.cc:302] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39e53e19894643dfb0ecbd7dd734207f; no voters:
I20260806 04:56:42.449286 3960 leader_election.cc:288] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:42.449400 3970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:42.450551 3970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [term 1 LEADER]: Becoming Leader. State: Replica: 39e53e19894643dfb0ecbd7dd734207f, State: Running, Role: LEADER
I20260806 04:56:42.451078 3970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e53e19894643dfb0ecbd7dd734207f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33857 } }
I20260806 04:56:42.451326 3960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:42.454058 3972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39e53e19894643dfb0ecbd7dd734207f. Latest consensus state: current_term: 1 leader_uuid: "39e53e19894643dfb0ecbd7dd734207f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e53e19894643dfb0ecbd7dd734207f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33857 } } }
I20260806 04:56:42.454414 3972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:42.454813 3971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39e53e19894643dfb0ecbd7dd734207f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e53e19894643dfb0ecbd7dd734207f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 33857 } } }
I20260806 04:56:42.455080 3971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:42.461932 3980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:42.467640 3980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:42.475358 3980 catalog_manager.cc:1347] Generated new cluster ID: e996e335dc914fe99cba1657c1303c3d
I20260806 04:56:42.475451 3980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:42.501999 3980 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:42.502414 3980 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:42.509212 3980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39e53e19894643dfb0ecbd7dd734207f: Generated new TSK 0
I20260806 04:56:42.510046 3980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:42.527743 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 3893
W20260806 04:56:42.528646 3955 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47575
W20260806 04:56:42.529265 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 3893: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:56:42.529395 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 3893
2026-08-06T04:56:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1758 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:42 dist-test-slave-1x4g krb5kdc[3992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:42 dist-test-slave-1x4g krb5kdc[3992](info): set up 2 sockets
Aug 06 04:56:42 dist-test-slave-1x4g krb5kdc[3992](info): commencing operation
krb5kdc: starting...
W20260806 04:56:43.532820 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:43 dist-test-slave-1x4g krb5kdc[3992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:56:43.773272 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41223
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:36097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:41223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:44.073187 4005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:44.073446 4005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:44.073534 4005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:44.079851 4005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:44.079931 4005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:44.079969 4005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:44.080014 4005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:44.080045 4005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:44.088395 4005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:41223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:44.089464 4005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:44.090586 4005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:44.101819 4011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:44.101854 4013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:44.101773 4010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:44.102933 4005 server_base.cc:1034] running on GCE node
I20260806 04:56:44.103670 4005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:44.104983 4005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:44.106209 4005 hybrid_clock.cc:648] HybridClock initialized: now 1785992204106166 us; error 48 us; skew 500 ppm
I20260806 04:56:44.106528 4005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:44 dist-test-slave-1x4g krb5kdc[3992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992204, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:44.109360 4005 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:56:44.110661 4005 webserver.cc:460] Webserver started at http://127.13.148.190:38663/ using document root <none> and password file <none>
I20260806 04:56:44.111351 4005 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:44.111493 4005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:44.111820 4005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:44.114630 4005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "b8e641ff2e684a9d977fb9bc205a433b"
format_stamp: "Formatted at 2026-08-06 04:56:44 on dist-test-slave-1x4g"
server_key: "b622524367b4334abcda8be3ffd0dff0"
server_key_iv: "9a08e11160d698b4bc05df576e6ab4a5"
server_key_version: "encryptionkey@0"
I20260806 04:56:44.115309 4005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "b8e641ff2e684a9d977fb9bc205a433b"
format_stamp: "Formatted at 2026-08-06 04:56:44 on dist-test-slave-1x4g"
server_key: "b622524367b4334abcda8be3ffd0dff0"
server_key_iv: "9a08e11160d698b4bc05df576e6ab4a5"
server_key_version: "encryptionkey@0"
I20260806 04:56:44.120139 4005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:56:44.123476 4020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:44.124661 4005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:56:44.124855 4005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "b8e641ff2e684a9d977fb9bc205a433b"
format_stamp: "Formatted at 2026-08-06 04:56:44 on dist-test-slave-1x4g"
server_key: "b622524367b4334abcda8be3ffd0dff0"
server_key_iv: "9a08e11160d698b4bc05df576e6ab4a5"
server_key_version: "encryptionkey@0"
I20260806 04:56:44.125023 4005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:44.142364 4005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:44.145772 4005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:44.146023 4005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:44.163487 4005 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:41223
I20260806 04:56:44.163493 4071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:41223 every 8 connection(s)
I20260806 04:56:44.164726 4005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:44.167994 4072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:44.174374 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 4005
I20260806 04:56:44.174523 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:44.174926 13906 external_mini_cluster.cc:1428] Setting key 9c0878694d9e196096f0a1c9d5faf5da
I20260806 04:56:44.178036 4072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b: Bootstrap starting.
I20260806 04:56:44.181763 4072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:44.183027 4072 log.cc:824] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:44.185454 4072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b: No bootstrap required, opened a new log
Aug 06 04:56:44 dist-test-slave-1x4g krb5kdc[3992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:56:44.194366 4072 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e641ff2e684a9d977fb9bc205a433b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41223 } }
I20260806 04:56:44.194706 4072 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:44.194814 4072 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e641ff2e684a9d977fb9bc205a433b, State: Initialized, Role: FOLLOWER
I20260806 04:56:44.195369 4072 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e641ff2e684a9d977fb9bc205a433b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41223 } }
I20260806 04:56:44.195585 4072 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:44.195726 4072 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:44.195899 4072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:44.198110 4072 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e641ff2e684a9d977fb9bc205a433b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41223 } }
I20260806 04:56:44.198562 4072 leader_election.cc:302] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8e641ff2e684a9d977fb9bc205a433b; no voters:
I20260806 04:56:44.199211 4072 leader_election.cc:288] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:44.199383 4077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:44.200443 4077 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [term 1 LEADER]: Becoming Leader. State: Replica: b8e641ff2e684a9d977fb9bc205a433b, State: Running, Role: LEADER
I20260806 04:56:44.201045 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:44.176509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45129 (local address 127.13.148.190:41223)
0806 04:56:44.177362 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:44.177383 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:44.177469 (+ 86us) server_negotiation.cc:407] Connection header received
0806 04:56:44.179876 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:44.179902 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:44.180362 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:44.180745 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:44.182122 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:44.183819 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:44.185004 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:44.185610 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:44.185772 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:44.190696 (+ 4924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:44.190706 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:44.190723 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:44.190801 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:44.195080 (+ 4279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:44.195956 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:44.195965 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:44.195972 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:44.196179 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:44.196708 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:44.196715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:44.196720 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:44.197522 (+ 802us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:44.198807 (+ 1285us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:44.199073 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 04:56:44.199563 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":376,"threads_started":1}
I20260806 04:56:44.201355 4077 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e641ff2e684a9d977fb9bc205a433b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41223 } }
I20260806 04:56:44.201584 4072 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:44.205231 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:44.210597 4079 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8e641ff2e684a9d977fb9bc205a433b. Latest consensus state: current_term: 1 leader_uuid: "b8e641ff2e684a9d977fb9bc205a433b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e641ff2e684a9d977fb9bc205a433b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41223 } } }
I20260806 04:56:44.211051 4079 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:44.211710 4078 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8e641ff2e684a9d977fb9bc205a433b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e641ff2e684a9d977fb9bc205a433b" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41223 } } }
I20260806 04:56:44.212086 4078 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:44.214591 4091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:44.224160 4091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:44.233031 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:44.209502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44923 (local address 127.13.148.190:41223)
0806 04:56:44.209692 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:44.209699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:44.216477 (+ 6778us) server_negotiation.cc:407] Connection header received
0806 04:56:44.216629 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:44.216634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:44.216705 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:44.216841 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:44.218705 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:44.219708 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:44.221494 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:44.222322 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:44.222447 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:44.225539 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:44.225548 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:44.225552 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:44.225586 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:44.229371 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:44.230303 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:44.230309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:44.230315 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:44.230460 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:44.230982 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:44.230989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:44.230993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:44.231442 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:44.231712 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:44.232380 (+ 668us) server_negotiation.cc:299] Negotiation successful
0806 04:56:44.232626 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:56:44.238654 4100 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 04:56:44.238713 4091 catalog_manager.cc:1347] Generated new cluster ID: 148fcf803b6244e8ac624426d7d8015d
I20260806 04:56:44.239185 4091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:44.275508 4091 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:44.276978 4091 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:44.283872 4091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8e641ff2e684a9d977fb9bc205a433b: Generated new TSK 0
I20260806 04:56:44.284631 4091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:44.300750 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 4005
I20260806 04:56:44.302415 4067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:44.405400 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 4005
2026-08-06T04:56:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1877 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:44 dist-test-slave-1x4g krb5kdc[4104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:44 dist-test-slave-1x4g krb5kdc[4104](info): set up 2 sockets
Aug 06 04:56:44 dist-test-slave-1x4g krb5kdc[4104](info): commencing operation
krb5kdc: starting...
W20260806 04:56:45.430089 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:45 dist-test-slave-1x4g krb5kdc[4104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:56:45.665786 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41673
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:35917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:41673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:45.969367 4117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:45.969667 4117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:45.969815 4117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:45.976378 4117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:45.976513 4117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:45.976615 4117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:45.976734 4117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:45.976831 4117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:45.985422 4117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:35917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:41673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:41673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:45.986536 4117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:45.987757 4117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:45.998718 4125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:45.998775 4123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:45.998754 4122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:45.999608 4117 server_base.cc:1034] running on GCE node
I20260806 04:56:46.000355 4117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:46.001627 4117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:46.002842 4117 hybrid_clock.cc:648] HybridClock initialized: now 1785992206002813 us; error 42 us; skew 500 ppm
I20260806 04:56:46.003156 4117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:46 dist-test-slave-1x4g krb5kdc[4104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992206, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:46.006027 4117 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:56:46.007292 4117 webserver.cc:460] Webserver started at http://127.13.148.190:44391/ using document root <none> and password file <none>
I20260806 04:56:46.007810 4117 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:46.007899 4117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:46.008118 4117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:46.010818 4117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "aeb35a5c8c574116a7589291a2d35f70"
format_stamp: "Formatted at 2026-08-06 04:56:46 on dist-test-slave-1x4g"
server_key: "41b9d28308962d7931139108f72eddfa"
server_key_iv: "3c3f189350e2937c12c766f3cbe3ff19"
server_key_version: "encryptionkey@0"
I20260806 04:56:46.011518 4117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "aeb35a5c8c574116a7589291a2d35f70"
format_stamp: "Formatted at 2026-08-06 04:56:46 on dist-test-slave-1x4g"
server_key: "41b9d28308962d7931139108f72eddfa"
server_key_iv: "3c3f189350e2937c12c766f3cbe3ff19"
server_key_version: "encryptionkey@0"
I20260806 04:56:46.016165 4117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:56:46.020005 4132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:46.021437 4117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:56:46.021602 4117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "aeb35a5c8c574116a7589291a2d35f70"
format_stamp: "Formatted at 2026-08-06 04:56:46 on dist-test-slave-1x4g"
server_key: "41b9d28308962d7931139108f72eddfa"
server_key_iv: "3c3f189350e2937c12c766f3cbe3ff19"
server_key_version: "encryptionkey@0"
I20260806 04:56:46.021791 4117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:46.040668 4117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:46.044332 4117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:46.044605 4117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:46.061757 4117 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:41673
I20260806 04:56:46.061791 4184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:41673 every 8 connection(s)
I20260806 04:56:46.062961 4117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:46.066279 4185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:46.067106 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 4117
I20260806 04:56:46.067276 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:46.067776 13906 external_mini_cluster.cc:1428] Setting key 6b93f8a922bc07531b39bb22dd04f7d0
I20260806 04:56:46.079452 4185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70: Bootstrap starting.
I20260806 04:56:46.081265 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:46.069381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41605 (local address 127.13.148.190:41673)
0806 04:56:46.070086 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:46.070096 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:46.070131 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:56:46.072412 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:46.072432 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:46.072807 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:46.073110 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:46.073667 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:46.075368 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:46.076506 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:46.077636 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:46.077819 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:46.078787 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:46.078797 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:46.078813 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:46.079403 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:46.079493 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:46.079522 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:46.079620 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:46.079812 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 04:56:46.080052 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":323,"threads_started":1}
I20260806 04:56:46.082777 4185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:46.083808 4185 log.cc:824] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:46.084481 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:46.086427 4185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70: No bootstrap required, opened a new log
I20260806 04:56:46.095227 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:46.089625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47891 (local address 127.13.148.190:41673)
0806 04:56:46.089794 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:46.089800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:46.090089 (+ 289us) server_negotiation.cc:407] Connection header received
0806 04:56:46.090236 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:46.090239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:46.090287 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:46.090355 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:46.090863 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:46.091563 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:46.093279 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:46.093639 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:46.093723 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:46.094656 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:46.094661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:46.094663 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:46.094677 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:46.094706 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:46.094718 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:46.094780 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:46.094911 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 04:56:46.094969 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 04:56:46.095636 4185 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb35a5c8c574116a7589291a2d35f70" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41673 } }
I20260806 04:56:46.095965 4185 raft_consensus.cc:374] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:46.096063 4185 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb35a5c8c574116a7589291a2d35f70, State: Initialized, Role: FOLLOWER
I20260806 04:56:46.096556 4185 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb35a5c8c574116a7589291a2d35f70" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41673 } }
I20260806 04:56:46.096838 4185 raft_consensus.cc:388] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:46.096971 4185 raft_consensus.cc:482] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:46.097136 4185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:46.098945 4185 raft_consensus.cc:504] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb35a5c8c574116a7589291a2d35f70" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41673 } }
I20260806 04:56:46.099359 4185 leader_election.cc:302] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aeb35a5c8c574116a7589291a2d35f70; no voters:
I20260806 04:56:46.099918 4185 leader_election.cc:288] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:46.099977 4195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:46.100809 4195 raft_consensus.cc:686] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [term 1 LEADER]: Becoming Leader. State: Replica: aeb35a5c8c574116a7589291a2d35f70, State: Running, Role: LEADER
I20260806 04:56:46.101369 4195 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb35a5c8c574116a7589291a2d35f70" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41673 } }
I20260806 04:56:46.102002 4185 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:46.105137 4196 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aeb35a5c8c574116a7589291a2d35f70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb35a5c8c574116a7589291a2d35f70" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41673 } } }
I20260806 04:56:46.105516 4196 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:46.105930 4197 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeb35a5c8c574116a7589291a2d35f70. Latest consensus state: current_term: 1 leader_uuid: "aeb35a5c8c574116a7589291a2d35f70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb35a5c8c574116a7589291a2d35f70" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 41673 } } }
I20260806 04:56:46.106324 4197 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:46.110358 4207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:46.114979 4207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:46.122898 4207 catalog_manager.cc:1347] Generated new cluster ID: 0e2b1369e47e4d7da4c6dfa41fc35478
I20260806 04:56:46.122989 4207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:46.130118 4207 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:46.131131 4207 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:46.141937 4207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aeb35a5c8c574116a7589291a2d35f70: Generated new TSK 0
I20260806 04:56:46.142627 4207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:46.156945 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 4117
W20260806 04:56:46.158149 4180 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41605
W20260806 04:56:46.158710 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 4117: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:56:46.158835 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 4117
2026-08-06T04:56:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1751 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:46 dist-test-slave-1x4g krb5kdc[4216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:46 dist-test-slave-1x4g krb5kdc[4216](info): set up 2 sockets
Aug 06 04:56:46 dist-test-slave-1x4g krb5kdc[4216](info): commencing operation
krb5kdc: starting...
W20260806 04:56:47.172606 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:47 dist-test-slave-1x4g krb5kdc[4216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:56:47.422170 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:38367
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:45827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:47.724742 4229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:47.725000 4229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:47.725142 4229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:47.731109 4229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:47.731204 4229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:47.731312 4229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:47.731426 4229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:47.731498 4229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:47.739704 4229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:45827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:38367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:38367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:47.740820 4229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:47.742053 4229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:47.752681 4234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:47.752709 4238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:47.752754 4235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:47.753736 4229 server_base.cc:1034] running on GCE node
I20260806 04:56:47.754346 4229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:47.755475 4229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:47.756716 4229 hybrid_clock.cc:648] HybridClock initialized: now 1785992207756683 us; error 34 us; skew 500 ppm
I20260806 04:56:47.757069 4229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:47 dist-test-slave-1x4g krb5kdc[4216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992207, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:47.765115 4229 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:56:47.766490 4229 webserver.cc:460] Webserver started at http://127.13.148.190:43253/ using document root <none> and password file <none>
I20260806 04:56:47.767127 4229 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:47.767279 4229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:47.767590 4229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:47.770409 4229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "3d76afb16b1f4ba38e5f5f1ba667ea8f"
format_stamp: "Formatted at 2026-08-06 04:56:47 on dist-test-slave-1x4g"
server_key: "556da4aaebefd2b6a984f3e54047afc2"
server_key_iv: "61c800bcc70f33238c6b5b2b9e7ce152"
server_key_version: "encryptionkey@0"
I20260806 04:56:47.771238 4229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "3d76afb16b1f4ba38e5f5f1ba667ea8f"
format_stamp: "Formatted at 2026-08-06 04:56:47 on dist-test-slave-1x4g"
server_key: "556da4aaebefd2b6a984f3e54047afc2"
server_key_iv: "61c800bcc70f33238c6b5b2b9e7ce152"
server_key_version: "encryptionkey@0"
I20260806 04:56:47.776309 4229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 04:56:47.779579 4245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:47.780900 4229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:56:47.781078 4229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "3d76afb16b1f4ba38e5f5f1ba667ea8f"
format_stamp: "Formatted at 2026-08-06 04:56:47 on dist-test-slave-1x4g"
server_key: "556da4aaebefd2b6a984f3e54047afc2"
server_key_iv: "61c800bcc70f33238c6b5b2b9e7ce152"
server_key_version: "encryptionkey@0"
I20260806 04:56:47.781277 4229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:47.786599 4229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:47.790350 4229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:47.790621 4229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:47.807977 4229 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:38367
I20260806 04:56:47.808022 4296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:38367 every 8 connection(s)
I20260806 04:56:47.809329 4229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:47.812769 4297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:47.814967 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 4229
I20260806 04:56:47.815122 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:47.815580 13906 external_mini_cluster.cc:1428] Setting key 7f478e80c1c5f89c83aed9cf6a6d85e8
I20260806 04:56:47.822572 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:47.817265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41075 (local address 127.13.148.190:38367)
0806 04:56:47.818104 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:47.818114 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:47.818152 (+ 38us) server_negotiation.cc:407] Connection header received
0806 04:56:47.819589 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:47.819607 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:47.819881 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:47.820149 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:47.821113 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:47.821119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:47.821131 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:47.821144 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:47.821204 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:47.821223 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:47.821388 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:47.821485 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 04:56:47.821672 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":417,"threads_started":1}
I20260806 04:56:47.823693 4297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f: Bootstrap starting.
I20260806 04:56:47.825812 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:47.826756 4297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:47.827910 4297 log.cc:824] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:47.830957 4297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f: No bootstrap required, opened a new log
I20260806 04:56:47.832942 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:47.830699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55627 (local address 127.13.148.190:38367)
0806 04:56:47.830842 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:47.830847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:47.831233 (+ 386us) server_negotiation.cc:407] Connection header received
0806 04:56:47.831349 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:47.831359 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:47.831420 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:47.831545 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:47.832357 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:47.832363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:47.832366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:56:47.832381 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:47.832418 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:56:47.832435 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:47.832550 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:47.832618 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 04:56:47.832664 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:56:47.838171 4297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d76afb16b1f4ba38e5f5f1ba667ea8f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38367 } }
I20260806 04:56:47.838529 4297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:56:47.838636 4297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d76afb16b1f4ba38e5f5f1ba667ea8f, State: Initialized, Role: FOLLOWER
I20260806 04:56:47.839254 4297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d76afb16b1f4ba38e5f5f1ba667ea8f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38367 } }
I20260806 04:56:47.839493 4297 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:56:47.839597 4297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:56:47.839712 4297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:47.841728 4297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d76afb16b1f4ba38e5f5f1ba667ea8f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38367 } }
I20260806 04:56:47.842141 4297 leader_election.cc:302] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d76afb16b1f4ba38e5f5f1ba667ea8f; no voters:
I20260806 04:56:47.842636 4297 leader_election.cc:288] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:56:47.842737 4307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:47.843372 4307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [term 1 LEADER]: Becoming Leader. State: Replica: 3d76afb16b1f4ba38e5f5f1ba667ea8f, State: Running, Role: LEADER
I20260806 04:56:47.843833 4307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d76afb16b1f4ba38e5f5f1ba667ea8f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38367 } }
I20260806 04:56:47.844429 4297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:47.846504 4308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d76afb16b1f4ba38e5f5f1ba667ea8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d76afb16b1f4ba38e5f5f1ba667ea8f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38367 } } }
I20260806 04:56:47.847151 4308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:47.847867 4309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d76afb16b1f4ba38e5f5f1ba667ea8f. Latest consensus state: current_term: 1 leader_uuid: "3d76afb16b1f4ba38e5f5f1ba667ea8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d76afb16b1f4ba38e5f5f1ba667ea8f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 38367 } } }
I20260806 04:56:47.848116 4309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:47.848935 4314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:47.853744 4314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:47.861856 4314 catalog_manager.cc:1347] Generated new cluster ID: ca9679e32c824bfb9a5a0fb3bd846070
I20260806 04:56:47.861976 4314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:47.871343 4314 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:47.871806 4314 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:47.882457 4314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d76afb16b1f4ba38e5f5f1ba667ea8f: Generated new TSK 0
I20260806 04:56:47.883118 4314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:47.894749 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 4229
W20260806 04:56:47.895654 4292 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41075
W20260806 04:56:47.896257 13906 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 4229: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 04:56:47.896380 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 4229
2026-08-06T04:56:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1737 ms)
[----------] 18 tests from AuthTokenIssuingTest (31984 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:47 dist-test-slave-1x4g krb5kdc[4328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:47 dist-test-slave-1x4g krb5kdc[4328](info): set up 2 sockets
Aug 06 04:56:47 dist-test-slave-1x4g krb5kdc[4328](info): commencing operation
krb5kdc: starting...
W20260806 04:56:48.921487 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:49 dist-test-slave-1x4g krb5kdc[4328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:56:49.165663 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:42347
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:42347,127.13.148.189:42599,127.13.148.188:43439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:49.472314 4342 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:56:49.472589 4342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:49.472644 4342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:49.472800 4342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:49.479035 4342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:49.479099 4342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:49.479166 4342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:49.479204 4342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:49.479308 4342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:49.488035 4342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:42347,127.13.148.189:42599,127.13.148.188:43439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:42347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:49.489146 4342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:49.490258 4342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:49.500838 4348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:49.500874 4350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:49.500916 4347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:49.501681 4342 server_base.cc:1034] running on GCE node
I20260806 04:56:49.502465 4342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:49.503649 4342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:49.504908 4342 hybrid_clock.cc:648] HybridClock initialized: now 1785992209504839 us; error 40 us; skew 500 ppm
I20260806 04:56:49.505223 4342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:49 dist-test-slave-1x4g krb5kdc[4328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992209, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:49.508024 4342 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:56:49.509356 4342 webserver.cc:460] Webserver started at http://127.13.148.190:32945/ using document root <none> and password file <none>
I20260806 04:56:49.510005 4342 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:49.510157 4342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:49.510485 4342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:49.513433 4342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "26314cf6ab9546fca7045561fef00413"
format_stamp: "Formatted at 2026-08-06 04:56:49 on dist-test-slave-1x4g"
server_key: "ebeab56c5ba1918329193eafefb74eeb"
server_key_iv: "bd84147a248ded8e6b3fea131d5e56d2"
server_key_version: "encryptionkey@0"
I20260806 04:56:49.514158 4342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "26314cf6ab9546fca7045561fef00413"
format_stamp: "Formatted at 2026-08-06 04:56:49 on dist-test-slave-1x4g"
server_key: "ebeab56c5ba1918329193eafefb74eeb"
server_key_iv: "bd84147a248ded8e6b3fea131d5e56d2"
server_key_version: "encryptionkey@0"
I20260806 04:56:49.518885 4342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:56:49.522038 4357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:49.523157 4342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:56:49.523319 4342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "26314cf6ab9546fca7045561fef00413"
format_stamp: "Formatted at 2026-08-06 04:56:49 on dist-test-slave-1x4g"
server_key: "ebeab56c5ba1918329193eafefb74eeb"
server_key_iv: "bd84147a248ded8e6b3fea131d5e56d2"
server_key_version: "encryptionkey@0"
I20260806 04:56:49.523514 4342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:49.551198 4342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:49.554760 4342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:49.555043 4342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:49.571836 4342 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:42347
I20260806 04:56:49.571872 4409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:42347 every 8 connection(s)
I20260806 04:56:49.573071 4342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:49.576259 4410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:49.578504 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 4342
I20260806 04:56:49.578711 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:49.579311 13906 external_mini_cluster.cc:1428] Setting key c1c09f46718bbba903331485c59d64c1
I20260806 04:56:49.582481 4410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:56:49 dist-test-slave-1x4g krb5kdc[4328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992209, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:56:49.602697 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:49.585215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42347 (local address 127.0.0.1:41400)
0806 04:56:49.585844 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:49.585883 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:49.587650 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:49.588350 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:49.588356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:49.589059 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:49.589370 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:49.589387 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:49.590786 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:49.590793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:49.591779 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:49.591791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:49.592683 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:49.592689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:49.592878 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:49.593431 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:49.593450 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:49.595275 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:49.599216 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:49.599226 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:49.599232 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:49.599582 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:49.600016 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:49.600019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:49.600021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:49.600149 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:49.601002 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:49.601007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:49.601162 (+ 155us) client_negotiation.cc:769] Sending connection context
0806 04:56:49.601361 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 04:56:49.601691 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":268,"threads_started":1}
I20260806 04:56:49.602730 4413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:49.586267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.13.148.190:42347)
0806 04:56:49.586875 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:49.586886 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:49.586944 (+ 58us) server_negotiation.cc:407] Connection header received
0806 04:56:49.587965 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:49.587970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:49.588153 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:49.588328 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:49.589588 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:49.590643 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:49.591923 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:49.592476 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:49.592648 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:49.595504 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:49.595515 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:49.595529 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:49.595565 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:49.599087 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:49.599719 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:49.599729 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:49.599734 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:49.599919 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:49.600250 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:49.600256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:49.600261 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:49.600886 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:49.601050 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:49.601560 (+ 510us) server_negotiation.cc:299] Negotiation successful
0806 04:56:49.601839 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":431,"thread_start_us":299,"threads_started":1}
I20260806 04:56:49.605495 4410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.148.189@KRBTEST.COM; defaulting to no policy
I20260806 04:56:49.606460 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:49.606152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:42599 (local address 127.0.0.1:44868)
0806 04:56:49.606283 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:49.606371 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 04:56:49.606750 4358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.189:42599: connect: Connection refused (error 111)
I20260806 04:56:49.607964 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:49.607686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:42599 (local address 127.0.0.1:44884)
0806 04:56:49.607794 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:49.607860 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 04:56:49.608167 4410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.189:42599: Network error: Client connection negotiation failed: client connection to 127.13.148.189:42599: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.189.keytab.
Entry for principal kudu/127.13.148.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.189.keytab.
I20260806 04:56:49.635239 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.189:42599
--webserver_interface=127.13.148.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.189.keytab
--principal=kudu/127.13.148.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:42347,127.13.148.189:42599,127.13.148.188:43439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:56:49.657419 4410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } attempt: 1
I20260806 04:56:49.658437 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:49.658053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:42599 (local address 127.0.0.1:44888)
0806 04:56:49.658209 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:49.658295 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 04:56:49.659266 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:49.658975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:42599 (local address 127.0.0.1:44900)
0806 04:56:49.659099 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:49.659160 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 04:56:49.659436 4410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.189:42599: Network error: Client connection negotiation failed: client connection to 127.13.148.189:42599: connect: Connection refused (error 111)
I20260806 04:56:49.727684 4410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } attempt: 2
I20260806 04:56:49.728740 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:49.728301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:42599 (local address 127.0.0.1:44914)
0806 04:56:49.728537 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:49.728617 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 04:56:49.729655 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:49.729325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:42599 (local address 127.0.0.1:44922)
0806 04:56:49.729475 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:49.729545 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 04:56:49.729830 4410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.189:42599: Network error: Client connection negotiation failed: client connection to 127.13.148.189:42599: connect: Connection refused (error 111)
I20260806 04:56:49.821058 4410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } attempt: 3
I20260806 04:56:49.822206 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:49.821780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:42599 (local address 127.0.0.1:44926)
0806 04:56:49.821973 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:49.822073 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 04:56:49.823154 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:49.822846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:42599 (local address 127.0.0.1:44936)
0806 04:56:49.822970 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:49.823037 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 04:56:49.823359 4410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.189:42599: Network error: Client connection negotiation failed: client connection to 127.13.148.189:42599: connect: Connection refused (error 111)
W20260806 04:56:49.927227 4417 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:56:49.927511 4417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:49.927614 4417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:49.927724 4417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:49.934401 4417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:49.934482 4417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:49.934549 4417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:49.934587 4417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:49.934685 4417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:49.943310 4417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:42347,127.13.148.189:42599,127.13.148.188:43439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.189.keytab
--principal=kudu/127.13.148.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.189:42599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.148.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:49.944401 4417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:49.945655 4417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:49.956182 4423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:49.956223 4425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:49.956147 4422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:49.957916 4417 server_base.cc:1034] running on GCE node
I20260806 04:56:49.958561 4417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:49.959848 4417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:49.961054 4417 hybrid_clock.cc:648] HybridClock initialized: now 1785992209961002 us; error 31 us; skew 500 ppm
I20260806 04:56:49.961390 4417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:49 dist-test-slave-1x4g krb5kdc[4328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992209, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:49.964248 4417 init.cc:377] Logged in from keytab as kudu/127.13.148.189@KRBTEST.COM (short username kudu)
I20260806 04:56:49.965687 4417 webserver.cc:460] Webserver started at http://127.13.148.189:43021/ using document root <none> and password file <none>
I20260806 04:56:49.966349 4417 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:49.966480 4417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:49.966620 4410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } attempt: 4
I20260806 04:56:49.966806 4417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:49.967621 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:49.967122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:42599 (local address 127.0.0.1:44944)
0806 04:56:49.967417 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:49.967497 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260806 04:56:49.968567 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:49.968216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:42599 (local address 127.0.0.1:44956)
0806 04:56:49.968369 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:49.968461 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 04:56:49.968757 4410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.189:42599: Network error: Client connection negotiation failed: client connection to 127.13.148.189:42599: connect: Connection refused (error 111)
I20260806 04:56:49.969772 4417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-1/data/instance:
uuid: "c4a36052415b4739a75803e27949b315"
format_stamp: "Formatted at 2026-08-06 04:56:49 on dist-test-slave-1x4g"
server_key: "aa3447e4968e4165a596f9a01af5104f"
server_key_iv: "2f1342a4865fec45ee18653425fd04ac"
server_key_version: "encryptionkey@0"
I20260806 04:56:49.970474 4417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-1/wal/instance:
uuid: "c4a36052415b4739a75803e27949b315"
format_stamp: "Formatted at 2026-08-06 04:56:49 on dist-test-slave-1x4g"
server_key: "aa3447e4968e4165a596f9a01af5104f"
server_key_iv: "2f1342a4865fec45ee18653425fd04ac"
server_key_version: "encryptionkey@0"
I20260806 04:56:49.975103 4417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:56:49.978271 4433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:49.979399 4417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:56:49.979563 4417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-1/wal
uuid: "c4a36052415b4739a75803e27949b315"
format_stamp: "Formatted at 2026-08-06 04:56:49 on dist-test-slave-1x4g"
server_key: "aa3447e4968e4165a596f9a01af5104f"
server_key_iv: "2f1342a4865fec45ee18653425fd04ac"
server_key_version: "encryptionkey@0"
I20260806 04:56:49.979751 4417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:49.996106 4417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:49.999668 4417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:49.999899 4417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:50.018167 4417 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.189:42599
I20260806 04:56:50.018169 4485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.189:42599 every 8 connection(s)
I20260806 04:56:50.019480 4417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-1/data/info.pb
I20260806 04:56:50.023279 4486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:50.024986 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 4417
I20260806 04:56:50.025169 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-1/wal/instance
I20260806 04:56:50.025725 13906 external_mini_cluster.cc:1428] Setting key 801e6dcebca46b4f8fbcd38a30df3a65
I20260806 04:56:50.032807 4486 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:56:50 dist-test-slave-1x4g krb5kdc[4328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992209, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.189@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:56:50.055915 4413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.036158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41406 (local address 127.13.148.190:42347)
0806 04:56:50.036372 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.036378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.037720 (+ 1342us) server_negotiation.cc:407] Connection header received
0806 04:56:50.040100 (+ 2380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.040104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.040176 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.040268 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.041869 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.043065 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.044615 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.045395 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.045563 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.049413 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:50.049418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:50.049422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:50.049473 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:50.052333 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.053378 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.053382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.053384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.053503 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.054020 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.054023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.054025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.054305 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:50.054436 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:50.055494 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0806 04:56:50.055687 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.13.148.188@KRBTEST.COM; defaulting to no policy
I20260806 04:56:50.057576 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.036572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42347 (local address 127.0.0.1:41406)
0806 04:56:50.037525 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.037577 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:50.039761 (+ 2184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:50.040482 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:50.040490 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:50.041099 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:50.041679 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:50.041697 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.043233 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.043237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:50.044461 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:50.044472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.045607 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.045612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:50.045810 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.046497 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:50.046518 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:50.049186 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:50.052500 (+ 3314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:50.052521 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:50.052542 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:50.053244 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:50.053622 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:50.053629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:50.053636 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:50.053904 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:50.054442 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:50.054453 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:50.055016 (+ 563us) client_negotiation.cc:769] Sending connection context
0806 04:56:50.055360 (+ 344us) client_negotiation.cc:240] Negotiation successful
0806 04:56:50.055957 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":374,"threads_started":1}
I20260806 04:56:50.059381 4486 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.188@KRBTEST.COM" created.
Aug 06 04:56:50 dist-test-slave-1x4g krb5kdc[4328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992209, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.189@KRBTEST.COM for kudu/127.13.148.189@KRBTEST.COM
I20260806 04:56:50.074702 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.060253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:42599 (local address 127.0.0.1:44972)
0806 04:56:50.060568 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.060589 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:50.060701 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:50.061438 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:50.061446 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:50.061976 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:50.062196 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:50.062211 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.063589 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.063596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:50.065048 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:50.065064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.065858 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.065869 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:50.066101 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.066764 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:50.066791 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:50.069291 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:50.072385 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:50.072389 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:50.072393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:50.072746 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:50.073127 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:50.073132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:50.073135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:50.073267 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:50.073789 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:50.073794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:50.073928 (+ 134us) client_negotiation.cc:769] Sending connection context
0806 04:56:50.074062 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 04:56:50.074318 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260806 04:56:50.074787 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.060336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44972 (local address 127.13.148.189:42599)
0806 04:56:50.060983 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.060988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.061015 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:56:50.061056 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.061062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.061253 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.061378 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.062379 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.063439 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.065187 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.065695 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.065812 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.069489 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:50.069501 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:50.069514 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:50.069539 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:50.072239 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.072897 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.072908 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.072910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.073029 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.073362 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.073365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.073368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.073682 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:50.073882 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:50.074199 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 04:56:50.074410 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":318,"threads_started":1}
I20260806 04:56:50.077462 4486 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:56:50.078639 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.078242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48488)
0806 04:56:50.078393 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.078491 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 04:56:50.078895 4435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
I20260806 04:56:50.080402 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.080116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48496)
0806 04:56:50.080240 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.080312 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 04:56:50.080591 4486 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:43439: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.188.keytab.
Entry for principal kudu/127.13.148.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.188.keytab.
I20260806 04:56:50.095963 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.188:43439
--webserver_interface=127.13.148.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.188.keytab
--principal=kudu/127.13.148.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:42347,127.13.148.189:42599,127.13.148.188:43439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:56:50.129892 4486 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } attempt: 1
I20260806 04:56:50.131081 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.130599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48502)
0806 04:56:50.130793 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.130925 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 04:56:50.132143 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.131779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48518)
0806 04:56:50.131964 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.132040 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 04:56:50.132373 4486 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:43439: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
I20260806 04:56:50.200644 4486 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } attempt: 2
I20260806 04:56:50.201855 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.201392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48530)
0806 04:56:50.201600 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.201699 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 04:56:50.202776 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.202500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48540)
0806 04:56:50.202625 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.202689 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 04:56:50.202971 4486 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:43439: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
I20260806 04:56:50.268147 4410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } attempt: 5
Aug 06 04:56:50 dist-test-slave-1x4g krb5kdc[4328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992209, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for kudu/127.13.148.189@KRBTEST.COM
I20260806 04:56:50.282166 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.269053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.13.148.189:42599)
0806 04:56:50.269204 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.269208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.269227 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:56:50.269466 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.269469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.269545 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.269627 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.270678 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.271365 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.273056 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.273468 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.273559 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.276956 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:50.276961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:50.276964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:50.276994 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:50.279537 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.280191 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.280197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.280199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.280331 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.280725 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.280729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.280731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.281053 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:50.281207 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:50.281600 (+ 393us) server_negotiation.cc:299] Negotiation successful
0806 04:56:50.281812 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 04:56:50.282145 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.268884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:42599 (local address 127.0.0.1:44986)
0806 04:56:50.269135 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.269161 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:50.269317 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:50.269719 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:50.269726 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:50.270297 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:50.270533 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:50.270546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.271498 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.271503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:50.272909 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:50.272923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.273626 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.273632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:50.273850 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.274461 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:50.274480 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:50.276706 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:50.279659 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:50.279663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:50.279667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:50.280059 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:50.280448 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:50.280453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:50.280456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:50.280601 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:50.281178 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:50.281183 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:50.281333 (+ 150us) client_negotiation.cc:769] Sending connection context
0806 04:56:50.281474 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 04:56:50.281749 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 04:56:50.283524 4410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:56:50.284600 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.284276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48544)
0806 04:56:50.284415 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.284484 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 04:56:50.285465 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.285178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48558)
0806 04:56:50.285310 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.285373 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 04:56:50.285660 4410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:43439: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
I20260806 04:56:50.294153 4486 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } attempt: 3
I20260806 04:56:50.294972 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.294608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48560)
0806 04:56:50.294757 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.294841 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 04:56:50.295840 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.295477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48566)
0806 04:56:50.295679 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.295744 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260806 04:56:50.296051 4486 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:43439: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
I20260806 04:56:50.336885 4410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } attempt: 1
I20260806 04:56:50.338037 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.337567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48582)
0806 04:56:50.337791 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.337884 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 04:56:50.339049 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.338714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48598)
0806 04:56:50.338874 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.338941 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 04:56:50.339249 4410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:43439: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
I20260806 04:56:50.407524 4410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } attempt: 2
I20260806 04:56:50.408576 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.408134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48614)
0806 04:56:50.408342 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.408424 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 04:56:50.409614 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.409299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48618)
0806 04:56:50.409449 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.409519 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 04:56:50.409826 4410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:43439: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
W20260806 04:56:50.411899 4493 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:56:50.412211 4493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:50.412310 4493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:50.412467 4493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:50.419021 4493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:50.419087 4493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:50.419134 4493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:50.419189 4493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:50.419273 4493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:50.427901 4493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:42347,127.13.148.189:42599,127.13.148.188:43439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.188.keytab
--principal=kudu/127.13.148.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.188:43439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.148.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:50.428964 4493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:50.430095 4493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:56:50.439255 4486 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } attempt: 4
I20260806 04:56:50.440460 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.439904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48630)
0806 04:56:50.440203 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.440299 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
W20260806 04:56:50.441519 4498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:50.441474 4501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:50.441884 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.441426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48632)
0806 04:56:50.441646 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.441739 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260806 04:56:50.442152 4486 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:43439: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43439: connect: Connection refused (error 111)
W20260806 04:56:50.442538 4499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:50.442579 4493 server_base.cc:1034] running on GCE node
I20260806 04:56:50.443203 4493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:50.444303 4493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:50.445534 4493 hybrid_clock.cc:648] HybridClock initialized: now 1785992210445484 us; error 29 us; skew 500 ppm
I20260806 04:56:50.445811 4493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:50 dist-test-slave-1x4g krb5kdc[4328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992210, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:50.448467 4493 init.cc:377] Logged in from keytab as kudu/127.13.148.188@KRBTEST.COM (short username kudu)
I20260806 04:56:50.449728 4493 webserver.cc:460] Webserver started at http://127.13.148.188:34063/ using document root <none> and password file <none>
I20260806 04:56:50.450301 4493 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:50.450421 4493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:50.450749 4493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:50.453501 4493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-2/data/instance:
uuid: "70d55a4ad8fd43dea2104e8832f8799e"
format_stamp: "Formatted at 2026-08-06 04:56:50 on dist-test-slave-1x4g"
server_key: "0bf2c841cef4c24df3a7e9e679f71f32"
server_key_iv: "b6e171a45ed080b966719bf9255fab59"
server_key_version: "encryptionkey@0"
I20260806 04:56:50.454181 4493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-2/wal/instance:
uuid: "70d55a4ad8fd43dea2104e8832f8799e"
format_stamp: "Formatted at 2026-08-06 04:56:50 on dist-test-slave-1x4g"
server_key: "0bf2c841cef4c24df3a7e9e679f71f32"
server_key_iv: "b6e171a45ed080b966719bf9255fab59"
server_key_version: "encryptionkey@0"
I20260806 04:56:50.458653 4493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:56:50.461563 4508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:50.462734 4493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:56:50.462895 4493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-2/wal
uuid: "70d55a4ad8fd43dea2104e8832f8799e"
format_stamp: "Formatted at 2026-08-06 04:56:50 on dist-test-slave-1x4g"
server_key: "0bf2c841cef4c24df3a7e9e679f71f32"
server_key_iv: "b6e171a45ed080b966719bf9255fab59"
server_key_version: "encryptionkey@0"
I20260806 04:56:50.463104 4493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:50.478940 4493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:50.482429 4493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:50.482666 4493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:50.499748 4493 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.188:43439
I20260806 04:56:50.499755 4560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.188:43439 every 8 connection(s)
I20260806 04:56:50.501024 4493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-2/data/info.pb
I20260806 04:56:50.501516 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 4493
I20260806 04:56:50.501710 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991872418975-13906-0/minicluster-data/master-2/wal/instance
I20260806 04:56:50.502331 13906 external_mini_cluster.cc:1428] Setting key 21d8e26be4dee867d98dc3cc53dd3518
I20260806 04:56:50.505182 4561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:50.511526 4561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:56:50 dist-test-slave-1x4g krb5kdc[4328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.188@KRBTEST.COM
I20260806 04:56:50.516028 4410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } attempt: 3
Aug 06 04:56:50 dist-test-slave-1x4g krb5kdc[4328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992210, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.188@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:56:50.524106 4565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.504014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.13.148.188:43439)
0806 04:56:50.504696 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.504707 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.504741 (+ 34us) server_negotiation.cc:407] Connection header received
0806 04:56:50.506471 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.506492 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.506832 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.507156 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.508480 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.509749 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.511005 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.511648 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.511879 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.516268 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:50.516279 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:50.516291 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:50.516341 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:50.519816 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.520645 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.520654 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.520663 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.520873 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.521311 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.521317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.521320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.521773 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:50.522098 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:50.522601 (+ 503us) server_negotiation.cc:299] Negotiation successful
0806 04:56:50.522970 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":302,"threads_started":1}
Aug 06 04:56:50 dist-test-slave-1x4g krb5kdc[4328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992209, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for kudu/127.13.148.188@KRBTEST.COM
I20260806 04:56:50.530602 4413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.513945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.13.148.190:42347)
0806 04:56:50.514167 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.514173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.514759 (+ 586us) server_negotiation.cc:407] Connection header received
0806 04:56:50.514984 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.514988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.515049 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.515170 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.516651 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.517698 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.519618 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.520091 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.520204 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.524021 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:50.524027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:50.524031 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:50.524060 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:50.527151 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.528222 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.528227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.528230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.528371 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.528750 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.528756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.528761 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.529185 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:50.529360 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:50.530015 (+ 655us) server_negotiation.cc:299] Negotiation successful
0806 04:56:50.530223 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:56:50.530630 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.513782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:42347 (local address 127.0.0.1:41410)
0806 04:56:50.514641 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.514675 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:50.514830 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:50.515282 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:50.515288 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:50.516057 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:50.516452 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:50.516465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.517873 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.517897 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:50.519460 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:50.519473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.520264 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.520269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:50.520459 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.521164 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:50.521193 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:50.523825 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:50.527335 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:50.527349 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:50.527357 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:50.528088 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:50.528506 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:50.528509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:50.528512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:50.528638 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:50.529472 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:50.529482 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:50.529657 (+ 175us) client_negotiation.cc:769] Sending connection context
0806 04:56:50.529881 (+ 224us) client_negotiation.cc:240] Negotiation successful
0806 04:56:50.530202 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":511,"threads_started":1}
I20260806 04:56:50.531992 4561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:56:50.532444 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.516540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48654)
0806 04:56:50.516848 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.516865 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:50.516978 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:50.518461 (+ 1483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:50.518466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:50.518939 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:50.519126 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:50.519136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.520566 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.520573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:50.522059 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:50.522075 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.523020 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.523025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:50.523229 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.523699 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:50.523714 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:50.526221 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:50.529544 (+ 3323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:50.529549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:50.529555 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:50.530152 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:50.530627 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:50.530635 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:50.530639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:50.530805 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:50.531465 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:50.531470 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:50.531650 (+ 180us) client_negotiation.cc:769] Sending connection context
0806 04:56:50.531790 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 04:56:50.532044 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260806 04:56:50.533514 4567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.516718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.13.148.188:43439)
0806 04:56:50.518112 (+ 1394us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.518117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.518159 (+ 42us) server_negotiation.cc:407] Connection header received
0806 04:56:50.518215 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.518219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.518298 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.518402 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.519293 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.520429 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.522199 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.522846 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.523009 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.526481 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:50.526487 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:50.526490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:50.526519 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:50.529381 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.530305 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.530313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.530317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.530472 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.530972 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.530978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.530984 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.531324 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:50.532846 (+ 1522us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:50.532966 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 04:56:50.533139 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":449,"threads_started":1}
Aug 06 04:56:50 dist-test-slave-1x4g krb5kdc[4328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:56:50.538774 4410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413: Bootstrap starting.
Aug 06 04:56:50 dist-test-slave-1x4g krb5kdc[4328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992210, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.188@KRBTEST.COM for kudu/127.13.148.189@KRBTEST.COM
I20260806 04:56:50.543100 4410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:50.544269 4410 log.cc:824] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:50.545926 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.527049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.13.148.190:42347)
0806 04:56:50.528993 (+ 1944us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.528996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.529012 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:56:50.529045 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.529048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.529088 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.529210 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.530895 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.532106 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.533840 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.534411 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.534530 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.539327 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:50.539333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:50.539336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:50.539384 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:50.543080 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.543846 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.543851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.543855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.543990 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.544368 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.544375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.544377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.544728 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:50.545067 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:50.545333 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 04:56:50.545547 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1841,"thread_start_us":1020,"threads_started":1}
I20260806 04:56:50.546715 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.532889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45002 (local address 127.13.148.189:42599)
0806 04:56:50.533066 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.533072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.533189 (+ 117us) server_negotiation.cc:407] Connection header received
0806 04:56:50.533334 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.533339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.533413 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.533512 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.534291 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.535190 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.536970 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.537619 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.537742 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.540939 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:50.540948 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:50.540957 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:50.540989 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:50.543924 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.544644 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.544649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.544653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.544841 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.545253 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.545258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.545260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.545551 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:50.545667 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:50.546177 (+ 510us) server_negotiation.cc:299] Negotiation successful
0806 04:56:50.546383 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:56:50.547451 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.532690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:42599 (local address 127.0.0.1:45002)
0806 04:56:50.533108 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.533123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:50.533237 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:50.533537 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:50.533541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:50.533989 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:50.534152 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:50.534163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.535347 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.535353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:50.536782 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:50.536830 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.537812 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.537818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:50.538031 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.538567 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:50.538593 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:50.540652 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:50.544035 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:50.544040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:50.544043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:50.544530 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:50.544977 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:50.544981 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:50.544983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:50.545137 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:50.545756 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:50.545759 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:50.545909 (+ 150us) client_negotiation.cc:769] Sending connection context
0806 04:56:50.546903 (+ 994us) client_negotiation.cc:240] Negotiation successful
0806 04:56:50.547132 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310}
I20260806 04:56:50.548350 4561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:56:50.548651 4410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413: No bootstrap required, opened a new log
Aug 06 04:56:50 dist-test-slave-1x4g krb5kdc[4328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992210, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.188@KRBTEST.COM for kudu/127.13.148.188@KRBTEST.COM
Aug 06 04:56:50 dist-test-slave-1x4g krb5kdc[4328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.189@KRBTEST.COM
I20260806 04:56:50.559064 4410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } }
I20260806 04:56:50.559432 4410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26314cf6ab9546fca7045561fef00413, State: Initialized, Role: FOLLOWER
I20260806 04:56:50.559834 4410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } }
I20260806 04:56:50.561090 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.549127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48662)
0806 04:56:50.549236 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.549253 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:50.549368 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:50.549650 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:50.549654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:50.549982 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:50.550096 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:50.550103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.550958 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.550963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:50.551881 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:50.551905 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.552649 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.552654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:50.552836 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.553328 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:50.553342 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:50.555156 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:50.558368 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:50.558372 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:50.558375 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:50.558814 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:50.559142 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:50.559146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:50.559149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:50.559285 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:50.559972 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:50.559978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:50.560162 (+ 184us) client_negotiation.cc:769] Sending connection context
0806 04:56:50.560328 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 04:56:50.560684 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260806 04:56:50.560858 4570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } } }
I20260806 04:56:50.561259 4570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:56:50.562021 4410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:50.562050 4567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.549061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.13.148.188:43439)
0806 04:56:50.549220 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.549226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.549305 (+ 79us) server_negotiation.cc:407] Connection header received
0806 04:56:50.549477 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.549481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.549550 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.549636 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.550216 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.550825 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.552054 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.552500 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.552636 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.555324 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:50.555328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:50.555331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:50.555355 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:50.558239 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.558919 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.558925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.558928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.559041 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.559394 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.559398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.559401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.559760 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:50.559887 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:50.561501 (+ 1614us) server_negotiation.cc:299] Negotiation successful
0806 04:56:50.561708 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:56:50.566259 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.548309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45004 (local address 127.13.148.189:42599)
0806 04:56:50.548454 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.548460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.548478 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:56:50.548564 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.548570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.548630 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.548736 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.550214 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.551183 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.552959 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.553534 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.553617 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.558744 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:50.558749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:50.558753 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:50.558782 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:50.562420 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.563241 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.563246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.563249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.563464 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.563967 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.563972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.563975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.564290 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:50.564589 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:50.565656 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0806 04:56:50.565867 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:56:50.567569 4561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e: Bootstrap starting.
I20260806 04:56:50.571866 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:50.572520 4561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:50.574131 4561 log.cc:824] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:50.575233 4385 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "26314cf6ab9546fca7045561fef00413"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41414
I20260806 04:56:50.575680 4385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:56:50.575906 4385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:50.580250 4385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } }
W20260806 04:56:50.582927 4581 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:56:50.583082 4581 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:56:50.585712 4385 leader_election.cc:288] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [CANDIDATE]: Term 1 election: Requested vote from peers c4a36052415b4739a75803e27949b315 (127.13.148.189:42599), 70d55a4ad8fd43dea2104e8832f8799e (127.13.148.188:43439)
I20260806 04:56:50.587515 4561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e: No bootstrap required, opened a new log
I20260806 04:56:50.595935 4561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } }
I20260806 04:56:50.596248 4561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d55a4ad8fd43dea2104e8832f8799e, State: Initialized, Role: FOLLOWER
I20260806 04:56:50.596702 4561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } }
I20260806 04:56:50.597621 4586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } } }
I20260806 04:56:50.598047 4586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:56:50.598665 4561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:50.598784 4535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26314cf6ab9546fca7045561fef00413" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "70d55a4ad8fd43dea2104e8832f8799e"
I20260806 04:56:50.599287 4535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:50.583181 4342 master_runner.cc:186] Error getting master registration for 127.13.148.189:42599: OK, instance_id { permanent_uuid: "c4a36052415b4739a75803e27949b315" instance_seqno: 1785992210012980 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 04:56:50.601588 4342 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:56:50.602773 4535 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26314cf6ab9546fca7045561fef00413 in term 1.
I20260806 04:56:50.603421 4359 leader_election.cc:302] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26314cf6ab9546fca7045561fef00413, 70d55a4ad8fd43dea2104e8832f8799e; no voters:
I20260806 04:56:50.603946 4570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:56:50.607928 4570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [term 1 LEADER]: Becoming Leader. State: Replica: 26314cf6ab9546fca7045561fef00413, State: Running, Role: LEADER
I20260806 04:56:50.608582 4570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } }
I20260806 04:56:50.618681 4592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26314cf6ab9546fca7045561fef00413. Latest consensus state: current_term: 1 leader_uuid: "26314cf6ab9546fca7045561fef00413" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } } }
I20260806 04:56:50.619057 4592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:50.629343 4599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:50.631310 4599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:50.634749 4605 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 04:56:50.646303 4535 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e [term 1 FOLLOWER]: Refusing update from remote peer 26314cf6ab9546fca7045561fef00413: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:56:50.647269 4570 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:56:50.650585 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.618399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41430 (local address 127.13.148.190:42347)
0806 04:56:50.618606 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.618613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.619545 (+ 932us) server_negotiation.cc:407] Connection header received
0806 04:56:50.619875 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.619882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.620053 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.620138 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.622830 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.624043 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.626203 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.626988 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.627092 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.639225 (+ 12133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:50.639235 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:50.639242 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:50.639298 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:50.645901 (+ 6603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.646764 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.646770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.646774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.646911 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.647429 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.647434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.647438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.647952 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:50.648186 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:50.649713 (+ 1527us) server_negotiation.cc:299] Negotiation successful
0806 04:56:50.650083 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:56:50.644481 4493 master_runner.cc:186] Error getting master registration for 127.13.148.189:42599: OK, instance_id { permanent_uuid: "c4a36052415b4739a75803e27949b315" instance_seqno: 1785992210012980 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 04:56:50.660445 4493 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:56:50.662675 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.619890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.13.148.189:42599)
0806 04:56:50.620191 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.620197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.639188 (+ 18991us) server_negotiation.cc:407] Connection header received
0806 04:56:50.639622 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.639630 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.639725 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.639806 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.641925 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.642849 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.644573 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.645265 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.645371 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.654584 (+ 9213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:50.654594 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:50.654601 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:50.654654 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:50.658747 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.659836 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.659841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.659844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.660005 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.660428 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.660433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.660436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.660847 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:50.661148 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:50.661876 (+ 728us) server_negotiation.cc:299] Negotiation successful
0806 04:56:50.662283 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260806 04:56:50.671598 4586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26314cf6ab9546fca7045561fef00413. Latest consensus state: current_term: 1 leader_uuid: "26314cf6ab9546fca7045561fef00413" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } } }
I20260806 04:56:50.671918 4586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:56:50.672231 4567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.628488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48672 (local address 127.13.148.188:43439)
0806 04:56:50.628678 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.628686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.649291 (+ 20605us) server_negotiation.cc:407] Connection header received
0806 04:56:50.649490 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.649494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.649578 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.649671 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.651170 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.652336 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.656722 (+ 4386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.657680 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.657831 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.665187 (+ 7356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:50.665194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:50.665200 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:50.665238 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:50.668904 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.669627 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.669632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.669635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.669804 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.670243 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.670247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.670250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.670571 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:50.670838 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:50.671226 (+ 388us) server_negotiation.cc:299] Negotiation successful
0806 04:56:50.671639 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:56:50.675630 4606 mvcc.cc:205] Tried to move back new op lower bound from 7315424094793043968 to 7315424094659162112. Current Snapshot: MvccSnapshot[applied={T|T < 7315424094793043968}]
I20260806 04:56:50.676234 4592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26314cf6ab9546fca7045561fef00413" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } } }
I20260806 04:56:50.676484 4592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:50.676890 4607 mvcc.cc:205] Tried to move back new op lower bound from 7315424094793043968 to 7315424094659162112. Current Snapshot: MvccSnapshot[applied={T|T < 7315424094793043968}]
I20260806 04:56:50.678825 4586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26314cf6ab9546fca7045561fef00413" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } } }
I20260806 04:56:50.679055 4586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:56:50.681181 4599 catalog_manager.cc:1347] Generated new cluster ID: 23ad3e39f7614497a4fcf471393ded27
I20260806 04:56:50.681300 4599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:50.697000 4599 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:50.698715 4599 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:50.707924 4599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413: Generated new TSK 0
I20260806 04:56:50.709086 4599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:56:50.741369 4486 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } attempt: 5
Aug 06 04:56:50 dist-test-slave-1x4g krb5kdc[4328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992209, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.189@KRBTEST.COM for kudu/127.13.148.188@KRBTEST.COM
I20260806 04:56:50.759912 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.742033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43439 (local address 127.0.0.1:48680)
0806 04:56:50.742447 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:50.742477 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:50.742589 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:50.743014 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:50.743020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:50.743524 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:50.743738 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:50.743752 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.746739 (+ 2987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.746746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:50.748104 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:50.748121 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.749072 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.749079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:50.749305 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.749966 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:50.749990 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:50.752978 (+ 2988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:50.756941 (+ 3963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:50.756959 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:50.756966 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:50.757477 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:50.757900 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:50.757907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:50.757912 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:50.758069 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:50.758811 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:50.758818 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:50.759028 (+ 210us) client_negotiation.cc:769] Sending connection context
0806 04:56:50.759169 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 04:56:50.759479 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274}
I20260806 04:56:50.760799 4567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.742441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48680 (local address 127.13.148.188:43439)
0806 04:56:50.742702 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.742708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.742728 (+ 20us) server_negotiation.cc:407] Connection header received
0806 04:56:50.742785 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.742790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.742862 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.742970 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.745644 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.746608 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.748252 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.748923 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.749038 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.753174 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:50.753182 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:50.753188 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:50.753227 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:50.756744 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.757615 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.757623 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.757629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.757776 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:50.758194 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:50.758200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:50.758205 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:50.758605 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:50.758742 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:50.760265 (+ 1523us) server_negotiation.cc:299] Negotiation successful
0806 04:56:50.760481 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260806 04:56:50.766731 4486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315: Bootstrap starting.
I20260806 04:56:50.770861 4486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:50.772281 4486 log.cc:824] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:50.776532 4486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315: No bootstrap required, opened a new log
I20260806 04:56:50.792706 4486 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } }
I20260806 04:56:50.793134 4486 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4a36052415b4739a75803e27949b315, State: Initialized, Role: FOLLOWER
I20260806 04:56:50.793613 4486 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } }
I20260806 04:56:50.794641 4616 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } } }
I20260806 04:56:50.795197 4616 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:56:50.795890 4486 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:50.795946 4461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26314cf6ab9546fca7045561fef00413" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c4a36052415b4739a75803e27949b315"
I20260806 04:56:50.796255 4460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:50.800716 4460 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315 [term 1 FOLLOWER]: Refusing update from remote peer 26314cf6ab9546fca7045561fef00413: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:56:50.802126 4570 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:56:50.808570 4461 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26314cf6ab9546fca7045561fef00413 in term 1.
W20260806 04:56:50.827517 4568 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:50.817250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.13.148.190:42347)
0806 04:56:50.817421 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.817427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.825327 (+ 7900us) server_negotiation.cc:407] Connection header received
0806 04:56:50.825638 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.825647 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.825759 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.825838 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.827202 (+ 1364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41434: BlockingRecv error: recv got EOF from 127.0.0.1:41434 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 04:56:50.829890 4567 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:50.819389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.13.148.188:43439)
0806 04:56:50.819543 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.819548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.828139 (+ 8591us) server_negotiation.cc:407] Connection header received
0806 04:56:50.828386 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.828394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.828516 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.828605 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.829608 (+ 1003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48682: BlockingRecv error: recv got EOF from 127.0.0.1:48682 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260806 04:56:50.831213 4490 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:50.819553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45018 (local address 127.13.148.189:42599)
0806 04:56:50.819756 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.819763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.829224 (+ 9461us) server_negotiation.cc:407] Connection header received
0806 04:56:50.829735 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.829741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.829836 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.829936 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.830976 (+ 1040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45018: BlockingRecv error: recv got EOF from 127.0.0.1:45018 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:56:50.857064 4616 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26314cf6ab9546fca7045561fef00413. Latest consensus state: current_term: 1 leader_uuid: "26314cf6ab9546fca7045561fef00413" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } } }
W20260806 04:56:50.857424 4490 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:50.848936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45022 (local address 127.13.148.189:42599)
0806 04:56:50.849132 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.849139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.855456 (+ 6317us) server_negotiation.cc:407] Connection header received
0806 04:56:50.855733 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.855738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.855856 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.855944 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.857064 (+ 1120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45022: BlockingRecv error: recv got EOF from 127.0.0.1:45022 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:56:50.857520 4616 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:56:50.857895 4639 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26314cf6ab9546fca7045561fef00413" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } } }
I20260806 04:56:50.860124 4639 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 04:56:50.874377 4490 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:50.868424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.13.148.189:42599)
0806 04:56:50.868592 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.868599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.873259 (+ 4660us) server_negotiation.cc:407] Connection header received
0806 04:56:50.873440 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.873445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.873514 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.873604 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.874234 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45026: BlockingRecv error: recv got EOF from 127.0.0.1:45026 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:56:50.879801 4642 catalog_manager.cc:1259] Loaded cluster ID: 23ad3e39f7614497a4fcf471393ded27
I20260806 04:56:50.879947 4642 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315: loading cluster ID for follower catalog manager: success
W20260806 04:56:50.881304 4490 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:50.879499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.13.148.189:42599)
0806 04:56:50.879643 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.879647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.879749 (+ 102us) server_negotiation.cc:407] Connection header received
0806 04:56:50.880015 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.880022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.880123 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.880210 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.881041 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45042: BlockingRecv error: recv got EOF from 127.0.0.1:45042 (error 108)
Metrics: {"server-negotiator.queue_time_us":55,"spinlock_wait_cycles":13568}
I20260806 04:56:50.883605 4642 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315: acquiring CA information for follower catalog manager: success
I20260806 04:56:50.885816 4642 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c4a36052415b4739a75803e27949b315: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:56:50.894210 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.887544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.13.148.189:42599)
0806 04:56:50.887716 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.887721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.888122 (+ 401us) server_negotiation.cc:407] Connection header received
0806 04:56:50.888244 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.888251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.888388 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.888491 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:50.888941 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.889881 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.891697 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.892451 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.892576 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.893068 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:50.893635 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:50.893763 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:50.893855 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 04:56:50.893942 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260806 04:56:50.911011 4567 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:50.906372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.13.148.188:43439)
0806 04:56:50.906555 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.906561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.909292 (+ 2731us) server_negotiation.cc:407] Connection header received
0806 04:56:50.909561 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.909567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.909684 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.909751 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.910789 (+ 1038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48684: BlockingRecv error: recv got EOF from 127.0.0.1:48684 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:56:50.927838 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.921634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.13.148.189:42599)
0806 04:56:50.921800 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.921805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.922329 (+ 524us) server_negotiation.cc:407] Connection header received
0806 04:56:50.922544 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.922549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.922641 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.922732 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:50.923104 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.924100 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.925771 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.926454 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.926575 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.927000 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:50.927131 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:50.927192 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:50.927417 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 04:56:50.927557 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 04:56:50.938855 4567 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:50.934763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.13.148.188:43439)
0806 04:56:50.934908 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.934912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.937356 (+ 2444us) server_negotiation.cc:407] Connection header received
0806 04:56:50.937554 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.937561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.937652 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.937750 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.938616 (+ 866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48686: BlockingRecv error: recv got EOF from 127.0.0.1:48686 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:56:50.964198 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:50.957774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.13.148.189:42599)
0806 04:56:50.957952 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.957958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.958417 (+ 459us) server_negotiation.cc:407] Connection header received
0806 04:56:50.958614 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.958620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.958704 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.958792 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:50.959265 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.960293 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.962157 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:50.962862 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:50.962997 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:50.963453 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:50.963578 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:50.963660 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:50.963826 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 04:56:50.963916 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 04:56:50.986958 4567 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:50.982167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.13.148.188:43439)
0806 04:56:50.982381 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:50.982389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:50.985420 (+ 3031us) server_negotiation.cc:407] Connection header received
0806 04:56:50.985642 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:50.985649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:50.985729 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:50.985792 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:50.986728 (+ 936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48696: BlockingRecv error: recv got EOF from 127.0.0.1:48696 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 04:56:51.056952 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:51.049656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.13.148.189:42599)
0806 04:56:51.049813 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:51.049819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:51.050818 (+ 999us) server_negotiation.cc:407] Connection header received
0806 04:56:51.051048 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:51.051053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:51.051139 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:51.051224 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:51.051754 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:51.053001 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:51.054836 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:51.055479 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:51.055604 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:51.056060 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:51.056203 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:51.056295 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:51.056467 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 04:56:51.056557 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 04:56:51.072497 4567 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:51.070571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.13.148.188:43439)
0806 04:56:51.070808 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:51.070813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:51.070903 (+ 90us) server_negotiation.cc:407] Connection header received
0806 04:56:51.071164 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:51.071171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:51.071259 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:51.071385 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:51.072260 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48710: BlockingRecv error: recv got EOF from 127.0.0.1:48710 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
I20260806 04:56:51.145871 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:51.140189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45082 (local address 127.13.148.189:42599)
0806 04:56:51.140451 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:51.140458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:51.140764 (+ 306us) server_negotiation.cc:407] Connection header received
0806 04:56:51.141002 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:51.141009 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:51.141112 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:51.141232 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:51.141637 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:51.142774 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:51.143964 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:51.144636 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:51.144797 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:51.145128 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:51.145267 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:51.145360 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:51.145470 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 04:56:51.145559 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260806 04:56:51.153882 4567 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:51.151650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48724 (local address 127.13.148.188:43439)
0806 04:56:51.151860 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:51.151867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:51.152396 (+ 529us) server_negotiation.cc:407] Connection header received
0806 04:56:51.152590 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:51.152595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:51.152663 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:51.152767 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:51.153680 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48724: BlockingRecv error: recv got EOF from 127.0.0.1:48724 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:56:51.289139 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:51.284929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45086 (local address 127.13.148.189:42599)
0806 04:56:51.285146 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:51.285153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:51.285428 (+ 275us) server_negotiation.cc:407] Connection header received
0806 04:56:51.285575 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:51.285579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:51.285643 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:51.285717 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:51.286059 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:51.286694 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:51.287798 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:51.288133 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:51.288214 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:51.288507 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:51.288598 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:51.288656 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:51.288760 (+ 104us) server_negotiation.cc:299] Negotiation successful
0806 04:56:51.288870 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260806 04:56:51.296250 4567 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:51.294617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48738 (local address 127.13.148.188:43439)
0806 04:56:51.294812 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:51.294818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:51.295008 (+ 190us) server_negotiation.cc:407] Connection header received
0806 04:56:51.295188 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:51.295192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:51.295254 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:51.295344 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:51.296038 (+ 694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48738: BlockingRecv error: recv got EOF from 127.0.0.1:48738 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:56:51.370182 4570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26314cf6ab9546fca7045561fef00413" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26314cf6ab9546fca7045561fef00413" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 42347 } } peers { permanent_uuid: "c4a36052415b4739a75803e27949b315" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 42599 } } peers { permanent_uuid: "70d55a4ad8fd43dea2104e8832f8799e" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43439 } } }
I20260806 04:56:51.370471 4570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26314cf6ab9546fca7045561fef00413 [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:51.560737 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:51.555910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45100 (local address 127.13.148.189:42599)
0806 04:56:51.556152 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:51.556159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:51.556331 (+ 172us) server_negotiation.cc:407] Connection header received
0806 04:56:51.556491 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:51.556495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:51.556567 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:51.556678 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:51.557090 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:51.557878 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:51.559221 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:51.559700 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:51.559794 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:51.560116 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:51.560211 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:51.560271 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:51.560383 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 04:56:51.560439 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260806 04:56:51.567510 4567 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:51.565763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48752 (local address 127.13.148.188:43439)
0806 04:56:51.565982 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:51.565990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:51.566230 (+ 240us) server_negotiation.cc:407] Connection header received
0806 04:56:51.566439 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:51.566444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:51.566521 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:51.566610 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:51.567332 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48752: BlockingRecv error: recv got EOF from 127.0.0.1:48752 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 04:56:51.639578 4605 catalog_manager.cc:1259] Loaded cluster ID: 23ad3e39f7614497a4fcf471393ded27
I20260806 04:56:51.639714 4605 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e: loading cluster ID for follower catalog manager: success
I20260806 04:56:51.642480 4605 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e: acquiring CA information for follower catalog manager: success
I20260806 04:56:51.644246 4605 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 70d55a4ad8fd43dea2104e8832f8799e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:56:52.087625 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:52.083076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45112 (local address 127.13.148.189:42599)
0806 04:56:52.083518 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:52.083522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:52.083701 (+ 179us) server_negotiation.cc:407] Connection header received
0806 04:56:52.083847 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:52.083851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:52.083915 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:52.084000 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:52.084327 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:52.085030 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:52.086246 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:52.086627 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:52.086715 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:52.087071 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:52.087163 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:52.087231 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:52.087353 (+ 122us) server_negotiation.cc:299] Negotiation successful
0806 04:56:52.087413 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":240,"threads_started":1}
I20260806 04:56:52.097466 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:52.092344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48758 (local address 127.13.148.188:43439)
0806 04:56:52.092823 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:52.092830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:52.092922 (+ 92us) server_negotiation.cc:407] Connection header received
0806 04:56:52.093047 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:52.093051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:52.093191 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:52.093284 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:52.093619 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:52.094232 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:52.095379 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:52.095762 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:52.095851 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:52.096333 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:52.096950 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:52.097073 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:52.097133 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 04:56:52.097196 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":266,"threads_started":1}
I20260806 04:56:52.109858 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:52.103031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.13.148.189:42599)
0806 04:56:52.103253 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:52.103262 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:52.104001 (+ 739us) server_negotiation.cc:407] Connection header received
0806 04:56:52.104349 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:52.104354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:52.104437 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:52.104504 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:52.105317 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:52.106207 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:52.108087 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:52.108450 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:52.108579 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:52.109122 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:52.109279 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:52.109376 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:52.109559 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 04:56:52.109652 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 04:56:52.111505 4747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:52.103031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.13.148.190:42347)
0806 04:56:52.104180 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:52.104187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:52.104731 (+ 544us) server_negotiation.cc:407] Connection header received
0806 04:56:52.105001 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:52.105008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:52.105230 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:52.105444 (+ 214us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:52.106059 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:52.106847 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:52.108754 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:52.109369 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:52.109546 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:52.110076 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:52.110572 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:52.110675 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:52.111070 (+ 395us) server_negotiation.cc:299] Negotiation successful
0806 04:56:52.111152 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":818,"threads_started":1}
I20260806 04:56:52.111696 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:52.104246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.13.148.188:43439)
0806 04:56:52.104409 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:52.104415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:52.105479 (+ 1064us) server_negotiation.cc:407] Connection header received
0806 04:56:52.105726 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:52.105732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:52.105811 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:52.105890 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:52.106420 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:52.107461 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:52.109479 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:52.110113 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:52.110215 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:52.110803 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:52.110916 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:52.110996 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:52.111227 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 04:56:52.111304 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:56:52.115857 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 4342
I20260806 04:56:52.117084 4405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:52.218555 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 4342
W20260806 04:56:52.240468 4512 connection.cc:535] server connection from 127.0.0.1:48654 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 04:56:52.240943 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 4417
I20260806 04:56:52.242048 4481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:52.341493 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 4417
I20260806 04:56:52.362150 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 4493
I20260806 04:56:52.363231 4556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:52.464138 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 4493
2026-08-06T04:56:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4572 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:56:52 dist-test-slave-1x4g krb5kdc[4756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:56:52 dist-test-slave-1x4g krb5kdc[4756](info): set up 2 sockets
Aug 06 04:56:52 dist-test-slave-1x4g krb5kdc[4756](info): commencing operation
krb5kdc: starting...
W20260806 04:56:53.492936 13906 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.002s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/test-user.keytab.
Aug 06 04:56:53 dist-test-slave-1x4g krb5kdc[4756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:56:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:56:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
Entry for principal kudu/127.13.148.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab.
I20260806 04:56:53.743036 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:36783
--webserver_interface=127.13.148.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:34629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:36783,127.13.148.189:38369,127.13.148.188:43391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:56:54.050573 4769 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:56:54.050892 4769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:54.050998 4769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:54.051133 4769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:54.057662 4769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:54.057777 4769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:54.057879 4769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:54.057955 4769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:54.058036 4769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:54.067188 4769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:34629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:36783,127.13.148.189:38369,127.13.148.188:43391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.190.keytab
--principal=kudu/127.13.148.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.190:36783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:54.068290 4769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:54.069533 4769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:54.079977 4774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:54.080060 4777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:54.080021 4775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:54.080956 4769 server_base.cc:1034] running on GCE node
I20260806 04:56:54.081645 4769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:54.082914 4769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:54.084148 4769 hybrid_clock.cc:648] HybridClock initialized: now 1785992214084098 us; error 41 us; skew 500 ppm
I20260806 04:56:54.084477 4769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:54 dist-test-slave-1x4g krb5kdc[4756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992214, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:54.092357 4769 init.cc:377] Logged in from keytab as kudu/127.13.148.190@KRBTEST.COM (short username kudu)
I20260806 04:56:54.093668 4769 webserver.cc:460] Webserver started at http://127.13.148.190:35249/ using document root <none> and password file <none>
I20260806 04:56:54.094262 4769 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:54.094364 4769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:54.094667 4769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:54.097373 4769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-0/data/instance:
uuid: "88e4b41a38af488fa5121819d8c5281f"
format_stamp: "Formatted at 2026-08-06 04:56:54 on dist-test-slave-1x4g"
server_key: "aa624cda307bcfd4ac32829c01903d4c"
server_key_iv: "6b6ed1372ad6f3be41fea5eee091ca2e"
server_key_version: "encryptionkey@0"
I20260806 04:56:54.098153 4769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-0/wal/instance:
uuid: "88e4b41a38af488fa5121819d8c5281f"
format_stamp: "Formatted at 2026-08-06 04:56:54 on dist-test-slave-1x4g"
server_key: "aa624cda307bcfd4ac32829c01903d4c"
server_key_iv: "6b6ed1372ad6f3be41fea5eee091ca2e"
server_key_version: "encryptionkey@0"
I20260806 04:56:54.102651 4769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:56:54.105773 4784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:54.106945 4769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260806 04:56:54.107110 4769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-0/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-0/wal
uuid: "88e4b41a38af488fa5121819d8c5281f"
format_stamp: "Formatted at 2026-08-06 04:56:54 on dist-test-slave-1x4g"
server_key: "aa624cda307bcfd4ac32829c01903d4c"
server_key_iv: "6b6ed1372ad6f3be41fea5eee091ca2e"
server_key_version: "encryptionkey@0"
I20260806 04:56:54.107293 4769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:54.121289 4769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:54.124933 4769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:54.125176 4769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:54.141580 4769 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.190:36783
I20260806 04:56:54.141600 4836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.190:36783 every 8 connection(s)
I20260806 04:56:54.142815 4769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-0/data/info.pb
I20260806 04:56:54.143819 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 4769
I20260806 04:56:54.143996 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-0/wal/instance
I20260806 04:56:54.144519 13906 external_mini_cluster.cc:1428] Setting key 804866f01a51e5fe8618a8b62bba1766
I20260806 04:56:54.146874 4837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:54.155828 4837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:56:54 dist-test-slave-1x4g krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992214, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.189@KRBTEST.COM" created.
I20260806 04:56:54.176537 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.159269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48494 (local address 127.13.148.190:36783)
0806 04:56:54.160050 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:54.160067 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:54.160432 (+ 365us) server_negotiation.cc:407] Connection header received
0806 04:56:54.162303 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:54.162309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:54.162524 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:54.162727 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:54.163842 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:54.165052 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:54.166219 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:54.166683 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:54.166867 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:54.170358 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:54.170365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:54.170374 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:54.170399 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:54.173174 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:54.173845 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:54.173857 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:54.173860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:54.174058 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:54.174409 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:54.174412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:54.174414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:54.174854 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:54.175025 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:54.175454 (+ 429us) server_negotiation.cc:299] Negotiation successful
0806 04:56:54.175716 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":551,"thread_start_us":362,"threads_started":1}
I20260806 04:56:54.177623 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.159276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:36783 (local address 127.0.0.1:48494)
0806 04:56:54.160292 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.160327 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:54.161954 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:54.162760 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:54.162765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:54.163352 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:54.163641 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:54.163656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:54.165208 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:54.165211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:54.166096 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:54.166106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:54.166903 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:54.166909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:54.167124 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:54.167716 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:54.167734 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:54.170140 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:54.173308 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:54.173318 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:54.173326 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:54.173706 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:54.174157 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:54.174160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:54.174162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:54.174312 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:54.174972 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:54.174976 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:54.175150 (+ 174us) client_negotiation.cc:769] Sending connection context
0806 04:56:54.175319 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 04:56:54.177087 (+ 1768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":1249,"thread_start_us":645,"threads_started":1}
I20260806 04:56:54.180117 4837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:56:54.181056 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.180614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:38369 (local address 127.0.0.1:33396)
0806 04:56:54.180799 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.180919 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:38369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 04:56:54.181303 4788 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.189:38369: connect: Connection refused (error 111)
I20260806 04:56:54.182890 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.182467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:38369 (local address 127.0.0.1:33412)
0806 04:56:54.182660 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.182755 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:38369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 04:56:54.183182 4837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.189:38369: Network error: Client connection negotiation failed: client connection to 127.13.148.189:38369: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.189.keytab.
Entry for principal kudu/127.13.148.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.189.keytab.
I20260806 04:56:54.204316 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.189:38369
--webserver_interface=127.13.148.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.189.keytab
--principal=kudu/127.13.148.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:34629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:36783,127.13.148.189:38369,127.13.148.188:43391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:56:54.232473 4837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } attempt: 1
I20260806 04:56:54.233673 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.233140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:38369 (local address 127.0.0.1:33422)
0806 04:56:54.233367 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.233504 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:38369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 04:56:54.234669 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.234304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:38369 (local address 127.0.0.1:33428)
0806 04:56:54.234440 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.234521 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:38369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 04:56:54.234844 4837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.189:38369: Network error: Client connection negotiation failed: client connection to 127.13.148.189:38369: connect: Connection refused (error 111)
I20260806 04:56:54.303088 4837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } attempt: 2
I20260806 04:56:54.304406 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.303761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:38369 (local address 127.0.0.1:33444)
0806 04:56:54.304058 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.304184 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:38369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 04:56:54.305586 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.305096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:38369 (local address 127.0.0.1:33450)
0806 04:56:54.305259 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.305388 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:38369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 04:56:54.305766 4837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.189:38369: Network error: Client connection negotiation failed: client connection to 127.13.148.189:38369: connect: Connection refused (error 111)
I20260806 04:56:54.397003 4837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } attempt: 3
I20260806 04:56:54.398298 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.397786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:38369 (local address 127.0.0.1:33452)
0806 04:56:54.398038 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.398142 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:38369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 04:56:54.399341 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.398923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:38369 (local address 127.0.0.1:33462)
0806 04:56:54.399094 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.399196 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:38369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 04:56:54.399552 4837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.189:38369: Network error: Client connection negotiation failed: client connection to 127.13.148.189:38369: connect: Connection refused (error 111)
W20260806 04:56:54.518568 4844 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:56:54.518826 4844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:54.518882 4844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:54.518947 4844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:54.525270 4844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:54.525344 4844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:54.525395 4844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:54.525441 4844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:54.525497 4844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:54.534497 4844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:34629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:36783,127.13.148.189:38369,127.13.148.188:43391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.189.keytab
--principal=kudu/127.13.148.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.189:38369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.148.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:54.535482 4844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:54.536626 4844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:56:54.542809 4837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } attempt: 4
I20260806 04:56:54.543903 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.543450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:38369 (local address 127.0.0.1:33466)
0806 04:56:54.543702 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.543781 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:38369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260806 04:56:54.544857 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.544476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:38369 (local address 127.0.0.1:33474)
0806 04:56:54.544624 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.544697 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.189:38369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 04:56:54.545049 4837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.189:38369: Network error: Client connection negotiation failed: client connection to 127.13.148.189:38369: connect: Connection refused (error 111)
W20260806 04:56:54.547542 4849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:54.547574 4850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:54.547513 4852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:54.548249 4844 server_base.cc:1034] running on GCE node
I20260806 04:56:54.548957 4844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:54.550122 4844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:54.551313 4844 hybrid_clock.cc:648] HybridClock initialized: now 1785992214551252 us; error 42 us; skew 500 ppm
I20260806 04:56:54.551642 4844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 04:56:54 dist-test-slave-1x4g krb5kdc[4756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992214, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:54.554703 4844 init.cc:377] Logged in from keytab as kudu/127.13.148.189@KRBTEST.COM (short username kudu)
I20260806 04:56:54.555981 4844 webserver.cc:460] Webserver started at http://127.13.148.189:40541/ using document root <none> and password file <none>
I20260806 04:56:54.556684 4844 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:54.556828 4844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:54.557148 4844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:54.560189 4844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-1/data/instance:
uuid: "b4469595beaf4b96a4ac50ecd34cb59f"
format_stamp: "Formatted at 2026-08-06 04:56:54 on dist-test-slave-1x4g"
server_key: "8d9359c35408d10d4b1877cc93695ac8"
server_key_iv: "94d1ac2f9654790e47fe217cc099c01f"
server_key_version: "encryptionkey@0"
I20260806 04:56:54.561105 4844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-1/wal/instance:
uuid: "b4469595beaf4b96a4ac50ecd34cb59f"
format_stamp: "Formatted at 2026-08-06 04:56:54 on dist-test-slave-1x4g"
server_key: "8d9359c35408d10d4b1877cc93695ac8"
server_key_iv: "94d1ac2f9654790e47fe217cc099c01f"
server_key_version: "encryptionkey@0"
I20260806 04:56:54.565708 4844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 04:56:54.568727 4860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:54.569888 4844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:56:54.570065 4844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-1/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-1/wal
uuid: "b4469595beaf4b96a4ac50ecd34cb59f"
format_stamp: "Formatted at 2026-08-06 04:56:54 on dist-test-slave-1x4g"
server_key: "8d9359c35408d10d4b1877cc93695ac8"
server_key_iv: "94d1ac2f9654790e47fe217cc099c01f"
server_key_version: "encryptionkey@0"
I20260806 04:56:54.570276 4844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:54.600508 4844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:54.604540 4844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:54.604809 4844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:54.622044 4844 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.189:38369
I20260806 04:56:54.622069 4913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.189:38369 every 8 connection(s)
I20260806 04:56:54.623288 4844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-1/data/info.pb
I20260806 04:56:54.626583 4914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:54.627815 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 4844
I20260806 04:56:54.628005 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-1/wal/instance
I20260806 04:56:54.628448 13906 external_mini_cluster.cc:1428] Setting key a7b973e97e22fb2761325de6b94370e2
I20260806 04:56:54.632750 4914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:56:54 dist-test-slave-1x4g krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992214, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.189@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:56:54.653463 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.635194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.13.148.190:36783)
0806 04:56:54.635398 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:54.635405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:54.636410 (+ 1005us) server_negotiation.cc:407] Connection header received
0806 04:56:54.639233 (+ 2823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:54.639237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:54.639304 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:54.639390 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:54.640847 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:54.642001 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:54.643412 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:54.644084 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:54.644210 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:54.647682 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:54.647687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:54.647690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:54.647721 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:54.650268 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:54.651152 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:54.651156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:54.651158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:54.651270 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:54.651735 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:54.651738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:54.651740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:54.651994 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:54.652099 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:54.652968 (+ 869us) server_negotiation.cc:299] Negotiation successful
0806 04:56:54.653164 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:56:54.654879 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.635562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:36783 (local address 127.0.0.1:48498)
0806 04:56:54.636236 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.636290 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:54.638963 (+ 2673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:54.639647 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:54.639660 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:54.640203 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:54.640667 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:54.640683 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:54.642152 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:54.642155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:54.643259 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:54.643270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:54.644257 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:54.644261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:54.644447 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:54.645109 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:54.645128 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:54.647485 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:54.650398 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:54.650408 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:54.650425 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:54.651042 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:54.651365 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:54.651372 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:54.651378 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:54.651625 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:54.652130 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:54.652141 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:54.652558 (+ 417us) client_negotiation.cc:769] Sending connection context
0806 04:56:54.652892 (+ 334us) client_negotiation.cc:240] Negotiation successful
0806 04:56:54.653440 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":251,"threads_started":1}
I20260806 04:56:54.656576 4914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.148.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.188@KRBTEST.COM" created.
Aug 06 04:56:54 dist-test-slave-1x4g krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992214, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.189@KRBTEST.COM for kudu/127.13.148.189@KRBTEST.COM
I20260806 04:56:54.672411 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.657317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:38369 (local address 127.0.0.1:33476)
0806 04:56:54.657459 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.657473 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:54.657575 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:54.658481 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:54.658485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:54.658978 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:54.659110 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:54.659119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:54.660653 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:54.660662 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:54.661584 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:54.661596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:54.662877 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:54.662881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:54.663045 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:54.663623 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:54.663637 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:54.665806 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:54.669942 (+ 4136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:54.669946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:54.669948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:54.670280 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:54.670698 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:54.670701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:54.670703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:54.670807 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:54.671511 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:54.671514 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:54.671638 (+ 124us) client_negotiation.cc:769] Sending connection context
0806 04:56:54.671741 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 04:56:54.671920 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":1}
I20260806 04:56:54.672433 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.657565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33476 (local address 127.13.148.189:38369)
0806 04:56:54.658104 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:54.658110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:54.658134 (+ 24us) server_negotiation.cc:407] Connection header received
0806 04:56:54.658185 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:54.658190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:54.658343 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:54.658487 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:54.659281 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:54.660480 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:54.661731 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:54.662722 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:54.662886 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:54.666015 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:54.666030 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:54.666049 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:54.666088 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:54.669803 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:54.670403 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:54.670415 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:54.670421 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:54.670571 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:54.670903 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:54.670910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:54.670915 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:54.671381 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:54.671633 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:54.671921 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 04:56:54.672148 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":401,"thread_start_us":263,"threads_started":1}
I20260806 04:56:54.675125 4914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:56:54.676121 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.675653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:48912)
0806 04:56:54.675852 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.675961 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 04:56:54.676427 4864 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
I20260806 04:56:54.677913 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.677622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:48926)
0806 04:56:54.677746 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.677822 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 04:56:54.678184 4914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:43391: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.188.keytab.
Entry for principal kudu/127.13.148.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.188.keytab.
I20260806 04:56:54.693827 13906 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10hk37/build/asan/bin/kudu
/tmp/dist-test-task10hk37/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.188:43391
--webserver_interface=127.13.148.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.188.keytab
--principal=kudu/127.13.148.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.148:34629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.190:36783,127.13.148.189:38369,127.13.148.188:43391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:56:54.727556 4914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } attempt: 1
I20260806 04:56:54.728601 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.728214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:48932)
0806 04:56:54.728388 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.728469 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 04:56:54.729604 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.729278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:48948)
0806 04:56:54.729432 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.729505 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 04:56:54.729801 4914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:43391: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
I20260806 04:56:54.798023 4914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } attempt: 2
I20260806 04:56:54.799062 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.798651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:48960)
0806 04:56:54.798828 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.798920 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 04:56:54.799961 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.799656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:48970)
0806 04:56:54.799787 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.799853 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 04:56:54.800154 4914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:43391: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
I20260806 04:56:54.844267 4837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } attempt: 5
Aug 06 04:56:54 dist-test-slave-1x4g krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992214, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for kudu/127.13.148.189@KRBTEST.COM
I20260806 04:56:54.862334 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.845030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:38369 (local address 127.0.0.1:33486)
0806 04:56:54.845231 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.845251 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:54.845376 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:54.845721 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:54.845725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:54.846183 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:54.846399 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:54.846411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:54.847750 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:54.847756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:54.848964 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:54.848980 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:54.849782 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:54.849787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:54.849981 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:54.854340 (+ 4359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:54.854369 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:54.856975 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:54.859841 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:54.859847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:54.859851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:54.860235 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:54.860636 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:54.860641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:54.860644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:54.860829 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:54.861336 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:54.861341 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:54.861497 (+ 156us) client_negotiation.cc:769] Sending connection context
0806 04:56:54.861656 (+ 159us) client_negotiation.cc:240] Negotiation successful
0806 04:56:54.861977 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 04:56:54.862334 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.845218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.13.148.189:38369)
0806 04:56:54.845388 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:54.845392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:54.845409 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:56:54.845504 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:54.845507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:54.845563 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:54.845639 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:54.846548 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:54.847600 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:54.849145 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:54.849639 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:54.850262 (+ 623us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:54.857158 (+ 6896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:54.857164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:54.857169 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:54.857208 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:54.859691 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:54.860387 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:54.860392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:54.860396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:54.860531 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:54.860943 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:54.860949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:54.860952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:54.861223 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:54.861377 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:54.861779 (+ 402us) server_negotiation.cc:299] Negotiation successful
0806 04:56:54.861992 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:56:54.863441 4837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:56:54.864427 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.864085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:48976)
0806 04:56:54.864233 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.864302 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 04:56:54.865257 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.864976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:48992)
0806 04:56:54.865098 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.865179 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 04:56:54.865466 4837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:43391: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
I20260806 04:56:54.891439 4914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } attempt: 3
I20260806 04:56:54.892536 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.892076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:48994)
0806 04:56:54.892296 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.892380 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 04:56:54.893499 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.893145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:48996)
0806 04:56:54.893339 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.893408 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260806 04:56:54.893687 4914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:43391: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
I20260806 04:56:54.916723 4837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } attempt: 1
I20260806 04:56:54.917814 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.917357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:49002)
0806 04:56:54.917549 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.917638 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 04:56:54.918766 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.918460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:49014)
0806 04:56:54.918608 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.918669 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 04:56:54.918953 4837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:43391: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
I20260806 04:56:54.987219 4837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } attempt: 2
I20260806 04:56:54.988245 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.987804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:49016)
0806 04:56:54.988015 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.988107 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 04:56:54.989187 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:54.988900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:49030)
0806 04:56:54.989028 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:54.989083 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 04:56:54.989365 4837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:43391: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
W20260806 04:56:55.001638 4921 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:56:55.001967 4921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:56:55.002095 4921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:56:55.002233 4921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:56:55.008826 4921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:56:55.008939 4921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:56:55.009058 4921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:56:55.009145 4921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:56:55.009222 4921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:56:55.018617 4921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.148:34629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.190:36783,127.13.148.189:38369,127.13.148.188:43391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/krb5kdc/kudu_127.13.148.188.keytab
--principal=kudu/127.13.148.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.188:43391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.148.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 03:56:12 UTC on ccycloud.quasar-guijbm.root.comops.site
build id 4569618
ASAN enabled
I20260806 04:56:55.020031 4921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:56:55.021359 4921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:56:55.031963 4930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:55.032032 4928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:56:55.032035 4927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:56:55.033041 4921 server_base.cc:1034] running on GCE node
I20260806 04:56:55.033665 4921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:56:55.034821 4921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:56:55.036039 4921 hybrid_clock.cc:648] HybridClock initialized: now 1785992215035976 us; error 43 us; skew 500 ppm
I20260806 04:56:55.036325 4921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:56:55.036875 4914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } attempt: 4
I20260806 04:56:55.037909 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.037423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:49044)
0806 04:56:55.037648 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:55.037731 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 04:56:55.038722 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.038410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:49048)
0806 04:56:55.038551 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:55.038621 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 04:56:55.038954 4914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:43391: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Aug 06 04:56:55 dist-test-slave-1x4g krb5kdc[4756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992215, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:56:55.039716 4921 init.cc:377] Logged in from keytab as kudu/127.13.148.188@KRBTEST.COM (short username kudu)
I20260806 04:56:55.041075 4921 webserver.cc:460] Webserver started at http://127.13.148.188:40303/ using document root <none> and password file <none>
I20260806 04:56:55.041728 4921 fs_manager.cc:359] Metadata directory not provided
I20260806 04:56:55.041857 4921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:56:55.042178 4921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:56:55.044831 4921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-2/data/instance:
uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8"
format_stamp: "Formatted at 2026-08-06 04:56:55 on dist-test-slave-1x4g"
server_key: "46f08a1096e612d876e82a5de9821a16"
server_key_iv: "4a92c1bcdcaa4c068974011523eef93f"
server_key_version: "encryptionkey@0"
I20260806 04:56:55.045531 4921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-2/wal/instance:
uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8"
format_stamp: "Formatted at 2026-08-06 04:56:55 on dist-test-slave-1x4g"
server_key: "46f08a1096e612d876e82a5de9821a16"
server_key_iv: "4a92c1bcdcaa4c068974011523eef93f"
server_key_version: "encryptionkey@0"
I20260806 04:56:55.050670 4921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 04:56:55.053779 4938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:56:55.054860 4921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:56:55.055047 4921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-2/data,/tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-2/wal
uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8"
format_stamp: "Formatted at 2026-08-06 04:56:55 on dist-test-slave-1x4g"
server_key: "46f08a1096e612d876e82a5de9821a16"
server_key_iv: "4a92c1bcdcaa4c068974011523eef93f"
server_key_version: "encryptionkey@0"
I20260806 04:56:55.055229 4921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:56:55.079586 4921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:56:55.083635 4921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:56:55.083881 4921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:56:55.095538 4837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } attempt: 3
I20260806 04:56:55.096560 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.096102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:49064)
0806 04:56:55.096310 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:55.096396 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 04:56:55.097812 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.097297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:49078)
0806 04:56:55.097532 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:55.097629 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260806 04:56:55.098044 4837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.188:43391: Network error: Client connection negotiation failed: client connection to 127.13.148.188:43391: connect: Connection refused (error 111)
I20260806 04:56:55.101184 4921 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.188:43391
I20260806 04:56:55.101159 4989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.188:43391 every 8 connection(s)
I20260806 04:56:55.102415 4921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-2/data/info.pb
I20260806 04:56:55.105721 4990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:56:55.109305 13906 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10hk37/build/asan/bin/kudu as pid 4921
I20260806 04:56:55.109500 13906 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10hk37/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991872418975-13906-0/minicluster-data/master-2/wal/instance
I20260806 04:56:55.110044 13906 external_mini_cluster.cc:1428] Setting key 6cdaa03abccc38f25cc20077c3a8303c
I20260806 04:56:55.113145 4990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:56:55 dist-test-slave-1x4g krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.188@KRBTEST.COM
Aug 06 04:56:55 dist-test-slave-1x4g krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992215, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.188@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:56:55.130752 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.116442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48506 (local address 127.13.148.190:36783)
0806 04:56:55.116618 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.116624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.117159 (+ 535us) server_negotiation.cc:407] Connection header received
0806 04:56:55.117387 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.117392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.117451 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.117560 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.118885 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.119620 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.121287 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.121708 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.121802 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:55.125199 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:55.125206 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:55.125213 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:55.125256 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:55.128106 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.128758 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.128762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.128764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.128938 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.129262 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.129266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.129269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.129575 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:55.129696 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:55.130258 (+ 562us) server_negotiation.cc:299] Negotiation successful
0806 04:56:55.130445 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:56:55.131480 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.116369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.190:36783 (local address 127.0.0.1:48506)
0806 04:56:55.117035 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:55.117068 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:55.117238 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:55.117624 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:55.117631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:55.118419 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:55.118726 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:55.118755 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.119773 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.119777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:55.121157 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:55.121172 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.121942 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.121954 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:55.122170 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:55.122791 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:55.122817 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:55.125021 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:55.128222 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:55.128232 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:55.128241 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:55.128650 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:55.129041 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:55.129044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:55.129046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:55.129155 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:55.129690 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:55.129698 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:55.129907 (+ 209us) client_negotiation.cc:769] Sending connection context
0806 04:56:55.130130 (+ 223us) client_negotiation.cc:240] Negotiation successful
0806 04:56:55.130592 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":141,"thread_start_us":279,"threads_started":1}
I20260806 04:56:55.131497 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.111789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.13.148.188:43391)
0806 04:56:55.112519 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.112530 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.112568 (+ 38us) server_negotiation.cc:407] Connection header received
0806 04:56:55.114091 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.114111 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.114467 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.114804 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.116241 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.117930 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.119383 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.120179 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.120289 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:55.124697 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:55.124703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:55.124712 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:55.124737 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:55.127749 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.128365 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.128370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.128372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.128508 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.128922 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.128926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.128928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.129475 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:55.129739 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:55.130056 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 04:56:55.130458 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":535,"thread_start_us":343,"threads_started":1}
I20260806 04:56:55.133309 4990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:56:55 dist-test-slave-1x4g krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992215, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.188@KRBTEST.COM for kudu/127.13.148.189@KRBTEST.COM
Aug 06 04:56:55 dist-test-slave-1x4g krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.190@KRBTEST.COM
I20260806 04:56:55.145027 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.133999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.189:38369 (local address 127.0.0.1:33496)
0806 04:56:55.134125 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:55.134145 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:55.134236 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:55.134704 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:55.134708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:55.135003 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:55.135106 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:55.135113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.135927 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.135933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:55.136689 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:55.136700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.137426 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.137432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:55.137606 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:55.138090 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:55.138103 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:55.139755 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:55.142964 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:55.142969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:55.142972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:55.143262 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:55.143637 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:55.143642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:55.143645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:55.143783 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:55.144244 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:55.144248 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:55.144372 (+ 124us) client_negotiation.cc:769] Sending connection context
0806 04:56:55.144488 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 04:56:55.144706 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 04:56:55.145097 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.134287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.13.148.189:38369)
0806 04:56:55.134479 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.134483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.134497 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:56:55.134539 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.134542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.134591 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.134683 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.135239 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.135809 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.136843 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.137281 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.137408 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:55.139918 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:55.139923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:55.139925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:55.139952 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:55.142853 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.143363 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.143368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.143372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.143524 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.143880 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.143883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.143884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.144107 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:55.144278 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:55.144585 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 04:56:55.144761 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:56:55.146045 4990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:56:55.148674 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.134351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48518 (local address 127.13.148.190:36783)
0806 04:56:55.134510 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.134515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.134604 (+ 89us) server_negotiation.cc:407] Connection header received
0806 04:56:55.134752 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.134760 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.134805 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.134889 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.135828 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.136440 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.137732 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.138128 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.138244 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:55.141850 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:55.141856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:55.141860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:55.141897 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:55.145464 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.146121 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.146125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.146128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.146268 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.146621 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.146625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.146628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.146925 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:55.147167 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:55.148308 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0806 04:56:55.148465 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 04:56:55 dist-test-slave-1x4g krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992215, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.188@KRBTEST.COM for kudu/127.13.148.188@KRBTEST.COM
I20260806 04:56:55.158476 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.146947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.13.148.188:43391)
0806 04:56:55.147066 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.147073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.147087 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:56:55.147124 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.147128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.147180 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.147275 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.148008 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.148992 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.150239 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.150691 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.150805 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:55.153568 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:55.153572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:55.153575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:55.153601 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:55.155906 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.156583 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.156586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.156588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.156720 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.157146 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.157149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.157151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.157368 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:55.157475 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:55.157958 (+ 483us) server_negotiation.cc:299] Negotiation successful
0806 04:56:55.158182 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":31}
I20260806 04:56:55.158692 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.146614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:49092)
0806 04:56:55.146741 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:55.146756 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:55.146838 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:55.147291 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:55.147295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:55.147698 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:55.147857 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:55.147867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.149142 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.149146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:55.150127 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:55.150138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.150810 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.150815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:55.151021 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:55.151496 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:55.151509 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:55.153411 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:55.156026 (+ 2615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:55.156030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:55.156033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:55.156474 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:55.156857 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:55.156863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:55.156868 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:55.157041 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:55.157486 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:55.157491 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:55.157696 (+ 205us) client_negotiation.cc:769] Sending connection context
0806 04:56:55.157852 (+ 156us) client_negotiation.cc:240] Negotiation successful
0806 04:56:55.158367 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":170}
Aug 06 04:56:55 dist-test-slave-1x4g krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.189@KRBTEST.COM
I20260806 04:56:55.162884 4990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8: Bootstrap starting.
I20260806 04:56:55.164350 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.150496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33500 (local address 127.13.148.189:38369)
0806 04:56:55.150929 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.150934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.150948 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:56:55.150990 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.150994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.151049 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.151136 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.152344 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.153192 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.154525 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.155108 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.155230 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:55.159678 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:55.159682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:55.159686 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:55.159721 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:55.162066 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.162679 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.162685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.162689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.162836 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.163206 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.163211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.163215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.163429 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:55.163674 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:55.163918 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 04:56:55.164109 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335}
I20260806 04:56:55.166033 4990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:55.167173 4990 log.cc:824] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:55.168556 13906 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:56:55.170053 4990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8: No bootstrap required, opened a new log
I20260806 04:56:55.171376 4812 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "88e4b41a38af488fa5121819d8c5281f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48518
I20260806 04:56:55.178617 4990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } }
I20260806 04:56:55.178942 4990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048c9b81ba774b9294bf5b4a0f4cc7b8, State: Initialized, Role: FOLLOWER
I20260806 04:56:55.179353 4990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } }
I20260806 04:56:55.180115 4996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } } }
I20260806 04:56:55.180408 4996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:56:55.181100 4990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 04:56:55.189046 5007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:56:55.189165 5007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:56:55.187189 4921 master_runner.cc:186] Error getting master registration for 127.13.148.190:36783: OK, instance_id { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" instance_seqno: 1785992214136950 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 04:56:55.197172 4921 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:56:55.275236 4837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } attempt: 4
Aug 06 04:56:55 dist-test-slave-1x4g krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992214, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.190@KRBTEST.COM for kudu/127.13.148.188@KRBTEST.COM
I20260806 04:56:55.290305 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.275934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:49096)
0806 04:56:55.276118 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:55.276138 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:55.276258 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:55.276662 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:55.276666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:55.277249 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:55.277458 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:55.277469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.278803 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.278807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:55.280313 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:55.280329 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.281314 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.281322 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:55.281568 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:55.282268 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:55.282292 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:55.284646 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:55.288087 (+ 3441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:55.288091 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:55.288095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:55.288526 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:55.288887 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:55.288891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:55.288894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:55.289043 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:55.289674 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:55.289677 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:55.289792 (+ 115us) client_negotiation.cc:769] Sending connection context
0806 04:56:55.289889 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 04:56:55.290065 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 04:56:55.290555 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.276011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.13.148.188:43391)
0806 04:56:55.276190 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.276195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.276218 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:56:55.276388 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.276398 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.276472 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.276606 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.277618 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.278660 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.280454 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.281102 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.281227 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:55.284820 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:55.284826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:55.284830 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:55.284865 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:55.287957 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.288635 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.288640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.288643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.288793 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.289142 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.289152 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.289156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.289573 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:55.289704 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:55.290012 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 04:56:55.290212 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 04:56:55.295713 4837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f: Bootstrap starting.
I20260806 04:56:55.299609 4837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:55.300765 4837 log.cc:824] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:55.303301 4837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f: No bootstrap required, opened a new log
I20260806 04:56:55.309702 4837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } }
I20260806 04:56:55.310055 4837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e4b41a38af488fa5121819d8c5281f, State: Initialized, Role: FOLLOWER
I20260806 04:56:55.310498 4837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } }
I20260806 04:56:55.311343 5010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } } }
I20260806 04:56:55.311764 5010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:56:55.312420 4837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:55.312541 4812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:56:55.312747 4812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:55.315896 4812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } }
I20260806 04:56:55.317342 4812 leader_election.cc:288] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [CANDIDATE]: Term 1 election: Requested vote from peers b4469595beaf4b96a4ac50ecd34cb59f (127.13.148.189:38369), 048c9b81ba774b9294bf5b4a0f4cc7b8 (127.13.148.188:43391)
I20260806 04:56:55.319340 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e4b41a38af488fa5121819d8c5281f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8"
I20260806 04:56:55.319828 4965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:55.323274 4965 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e4b41a38af488fa5121819d8c5281f in term 1.
I20260806 04:56:55.323844 4788 leader_election.cc:302] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048c9b81ba774b9294bf5b4a0f4cc7b8, 88e4b41a38af488fa5121819d8c5281f; no voters:
I20260806 04:56:55.324350 5010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [term 1 FOLLOWER]: Leader election won for term 1
W20260806 04:56:55.335681 4944 tablet.cc:2369] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:56:55.338153 4914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } attempt: 5
I20260806 04:56:55.338706 5010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [term 1 LEADER]: Becoming Leader. State: Replica: 88e4b41a38af488fa5121819d8c5281f, State: Running, Role: LEADER
I20260806 04:56:55.339594 5010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } }
Aug 06 04:56:55 dist-test-slave-1x4g krb5kdc[4756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785992214, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.189@KRBTEST.COM for kudu/127.13.148.188@KRBTEST.COM
I20260806 04:56:55.352172 5021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e4b41a38af488fa5121819d8c5281f. Latest consensus state: current_term: 1 leader_uuid: "88e4b41a38af488fa5121819d8c5281f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } } }
I20260806 04:56:55.353010 5021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:55.356240 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.338720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.188:43391 (local address 127.0.0.1:49098)
0806 04:56:55.339059 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0806 04:56:55.339083 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 04:56:55.339192 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:56:55.340047 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:56:55.340053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:56:55.340532 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:56:55.340727 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:55.340739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.342109 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.342115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:55.343433 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:56:55.343448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.344737 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.344744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:56:55.345088 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:55.345763 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:56:55.345791 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:56:55.348811 (+ 3020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:56:55.353161 (+ 4350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:55.353166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:55.353169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:55.353633 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:55.354131 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:56:55.354138 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:56:55.354144 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:56:55.354360 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:56:55.355110 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:56:55.355118 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:56:55.355299 (+ 181us) client_negotiation.cc:769] Sending connection context
0806 04:56:55.355454 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 04:56:55.355767 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260806 04:56:55.357338 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.339384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.13.148.188:43391)
0806 04:56:55.339543 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.339551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.339569 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:56:55.339621 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.339627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.339726 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.339819 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.340902 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.341934 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.343580 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.344280 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.344458 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:55.349039 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:55.349048 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:55.349055 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:55.349095 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:55.353021 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.353782 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.353789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.353794 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.353962 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.354499 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.354507 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.354512 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.354882 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:55.355045 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:55.356589 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0806 04:56:55.356895 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 04:56:55.361486 5028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:56:55.363250 4914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f: Bootstrap starting.
I20260806 04:56:55.367566 4914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f: Neither blocks nor log segments found. Creating new log.
I20260806 04:56:55.369341 4914 log.cc:824] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f: Log is configured to *not* fsync() on all Append() calls
I20260806 04:56:55.377153 4914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f: No bootstrap required, opened a new log
W20260806 04:56:55.377549 4790 tablet.cc:2369] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:56:55.379523 5028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:56:55.389534 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.340610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48522 (local address 127.13.148.190:36783)
0806 04:56:55.340752 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.340759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.357329 (+ 16570us) server_negotiation.cc:407] Connection header received
0806 04:56:55.357621 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.357627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.357703 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.357784 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.360266 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.361591 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.364013 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.364728 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.364869 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:55.380961 (+ 16092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:55.380969 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:55.380976 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:55.381035 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:55.385715 (+ 4680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.386483 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.386488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.386491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.386666 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.387077 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.387081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.387084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.387419 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:55.388430 (+ 1011us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:55.388553 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 04:56:55.388793 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:56:55.395682 4914 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } }
I20260806 04:56:55.396075 4914 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4469595beaf4b96a4ac50ecd34cb59f, State: Initialized, Role: FOLLOWER
I20260806 04:56:55.396549 4914 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } }
I20260806 04:56:55.403007 5036 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 04:56:55.403712 5038 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } } }
I20260806 04:56:55.404863 5038 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:56:55.408743 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.357588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33516 (local address 127.13.148.189:38369)
0806 04:56:55.357748 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.357767 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.364605 (+ 6838us) server_negotiation.cc:407] Connection header received
0806 04:56:55.378457 (+ 13852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.378463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.378537 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.378597 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.384632 (+ 6035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.385620 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.389793 (+ 4173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.390385 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.390471 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:55.397552 (+ 7081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:55.397559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:55.397564 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:55.397601 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:55.400715 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.405849 (+ 5134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.405855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.405857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.406016 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.406440 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.406444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.406447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.406785 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:55.407023 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:55.408159 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0806 04:56:55.408403 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:56:55.409642 4914 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:56:55.409696 4887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e4b41a38af488fa5121819d8c5281f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4469595beaf4b96a4ac50ecd34cb59f"
I20260806 04:56:55.410148 4887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:56:55.412911 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:55.365013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49106 (local address 127.13.148.188:43391)
0806 04:56:55.369067 (+ 4054us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.369075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.380493 (+ 11418us) server_negotiation.cc:407] Connection header received
0806 04:56:55.381325 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.381330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.381406 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.381468 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.387263 (+ 5795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.388103 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.391647 (+ 3544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:55.392349 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:55.392454 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:55.401569 (+ 9115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:56:55.401575 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:56:55.401578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:56:55.401633 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:56:55.405056 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.407455 (+ 2399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.407459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.407462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.407596 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:56:55.410886 (+ 3290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:56:55.410891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:56:55.410894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:56:55.411251 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:56:55.411496 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:55.412222 (+ 726us) server_negotiation.cc:299] Negotiation successful
0806 04:56:55.412469 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3907,"thread_start_us":19432,"threads_started":1}
I20260806 04:56:55.414067 4965 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8 [term 1 FOLLOWER]: Refusing update from remote peer 88e4b41a38af488fa5121819d8c5281f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:56:55.414893 5021 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [LEADER]: Connected to new peer: Peer: permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:56:55.415680 4887 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e4b41a38af488fa5121819d8c5281f in term 1.
I20260806 04:56:55.416569 4886 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f [term 1 FOLLOWER]: Refusing update from remote peer 88e4b41a38af488fa5121819d8c5281f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:56:55.418608 5021 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:56:55.407847 4769 master_runner.cc:186] Error getting master registration for 127.13.148.189:38369: OK, instance_id { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" instance_seqno: 1785992214617213 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 04:56:55.424110 4769 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:56:55.443547 4996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e4b41a38af488fa5121819d8c5281f. Latest consensus state: current_term: 1 leader_uuid: "88e4b41a38af488fa5121819d8c5281f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } } }
I20260806 04:56:55.443887 4996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:56:55.446707 5037 mvcc.cc:205] Tried to move back new op lower bound from 7315424114306641920 to 7315424114039640064. Current Snapshot: MvccSnapshot[applied={T|T < 7315424114306641920}]
I20260806 04:56:55.447417 5040 mvcc.cc:205] Tried to move back new op lower bound from 7315424114306641920 to 7315424114039640064. Current Snapshot: MvccSnapshot[applied={T|T < 7315424114306641920}]
I20260806 04:56:55.448098 4996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88e4b41a38af488fa5121819d8c5281f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } } }
I20260806 04:56:55.448290 4996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:56:55.454689 5021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88e4b41a38af488fa5121819d8c5281f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } } }
I20260806 04:56:55.454985 5021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:55.455358 5028 catalog_manager.cc:1347] Generated new cluster ID: 8dfa27e0d1fc4afb8b948968561a7c1d
I20260806 04:56:55.455461 5028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:56:55.475419 5038 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e4b41a38af488fa5121819d8c5281f. Latest consensus state: current_term: 1 leader_uuid: "88e4b41a38af488fa5121819d8c5281f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } } }
I20260806 04:56:55.475778 5038 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:56:55.478256 5053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88e4b41a38af488fa5121819d8c5281f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } } }
I20260806 04:56:55.478675 5053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f [sys.catalog]: This master's current role is: LEADER
I20260806 04:56:55.483987 5038 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88e4b41a38af488fa5121819d8c5281f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e4b41a38af488fa5121819d8c5281f" member_type: VOTER last_known_addr { host: "127.13.148.190" port: 36783 } } peers { permanent_uuid: "b4469595beaf4b96a4ac50ecd34cb59f" member_type: VOTER last_known_addr { host: "127.13.148.189" port: 38369 } } peers { permanent_uuid: "048c9b81ba774b9294bf5b4a0f4cc7b8" member_type: VOTER last_known_addr { host: "127.13.148.188" port: 43391 } } }
I20260806 04:56:55.484210 5038 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:56:55.490374 5041 mvcc.cc:205] Tried to move back new op lower bound from 7315424114306641920 to 7315424114039640064. Current Snapshot: MvccSnapshot[applied={T|T < 7315424114306641920 or (T in {7315424114306641920})}]
I20260806 04:56:55.501730 5057 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 04:56:55.524914 5028 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:56:55.526162 5028 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:56:55.539481 5028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88e4b41a38af488fa5121819d8c5281f: Generated new TSK 0
I20260806 04:56:55.540566 5028 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 04:56:55.607161 4840 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:55.604313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48530 (local address 127.13.148.190:36783)
0806 04:56:55.604576 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.604585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.605110 (+ 525us) server_negotiation.cc:407] Connection header received
0806 04:56:55.605394 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.605401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.605484 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.605581 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.606920 (+ 1339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48530: BlockingRecv error: recv got EOF from 127.0.0.1:48530 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260806 04:56:55.609777 4918 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:55.605242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33520 (local address 127.13.148.189:38369)
0806 04:56:55.605438 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.605444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.607806 (+ 2362us) server_negotiation.cc:407] Connection header received
0806 04:56:55.608033 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.608041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.608148 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.608230 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.609501 (+ 1271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33520: BlockingRecv error: recv got EOF from 127.0.0.1:33520 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 04:56:55.612910 4993 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:55.611027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49114 (local address 127.13.148.188:43391)
0806 04:56:55.611190 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.611195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.611352 (+ 157us) server_negotiation.cc:407] Connection header received
0806 04:56:55.611592 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.611615 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.611691 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.611752 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.612619 (+ 867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49114: BlockingRecv error: recv got EOF from 127.0.0.1:49114 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 04:56:55.622457 4918 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:55.616448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.13.148.189:38369)
0806 04:56:55.616622 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.616629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.620934 (+ 4305us) server_negotiation.cc:407] Connection header received
0806 04:56:55.621162 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.621166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.621252 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.621344 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.622178 (+ 834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33532: BlockingRecv error: recv got EOF from 127.0.0.1:33532 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 04:56:55.654649 4918 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:55.649775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.13.148.189:38369)
0806 04:56:55.649989 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.649995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.653351 (+ 3356us) server_negotiation.cc:407] Connection header received
0806 04:56:55.653515 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.653521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.653604 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.653705 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.654474 (+ 769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33536: BlockingRecv error: recv got EOF from 127.0.0.1:33536 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 04:56:55.664402 4918 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:55.662307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.13.148.189:38369)
0806 04:56:55.662463 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.662469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.663013 (+ 544us) server_negotiation.cc:407] Connection header received
0806 04:56:55.663170 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.663174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.663260 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.663361 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.664196 (+ 835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33548: BlockingRecv error: recv got EOF from 127.0.0.1:33548 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 04:56:55.673019 4918 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:55.671099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.13.148.189:38369)
0806 04:56:55.671260 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.671265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.671726 (+ 461us) server_negotiation.cc:407] Connection header received
0806 04:56:55.671874 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.671881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.671971 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.672074 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.672836 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33554: BlockingRecv error: recv got EOF from 127.0.0.1:33554 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260806 04:56:55.685173 4918 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:55.683584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.13.148.189:38369)
0806 04:56:55.683721 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.683728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.684079 (+ 351us) server_negotiation.cc:407] Connection header received
0806 04:56:55.684262 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.684266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.684332 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.684419 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.685003 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33558: BlockingRecv error: recv got EOF from 127.0.0.1:33558 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260806 04:56:55.705838 4918 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:55.704175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.13.148.189:38369)
0806 04:56:55.704368 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.704373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.704660 (+ 287us) server_negotiation.cc:407] Connection header received
0806 04:56:55.704842 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.704852 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.704905 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.704972 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.705638 (+ 666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33568: BlockingRecv error: recv got EOF from 127.0.0.1:33568 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 04:56:55.743603 4918 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:55.741549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.13.148.189:38369)
0806 04:56:55.741830 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.741836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.742160 (+ 324us) server_negotiation.cc:407] Connection header received
0806 04:56:55.742372 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.742376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.742458 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.742549 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.743345 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33572: BlockingRecv error: recv got EOF from 127.0.0.1:33572 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260806 04:56:55.813289 4918 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:55.811227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.13.148.189:38369)
0806 04:56:55.811439 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.811446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.811787 (+ 341us) server_negotiation.cc:407] Connection header received
0806 04:56:55.811958 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.811964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.812068 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.812199 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.813073 (+ 874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33576: BlockingRecv error: recv got EOF from 127.0.0.1:33576 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 04:56:55.947257 4918 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:55.945550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.13.148.189:38369)
0806 04:56:55.945733 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:55.945737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:55.946098 (+ 361us) server_negotiation.cc:407] Connection header received
0806 04:56:55.946250 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:55.946254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:55.946321 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:55.946413 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:55.947027 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33578: BlockingRecv error: recv got EOF from 127.0.0.1:33578 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:56:56.190964 5007 catalog_manager.cc:1259] Loaded cluster ID: 8dfa27e0d1fc4afb8b948968561a7c1d
I20260806 04:56:56.191063 5007 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8: loading cluster ID for follower catalog manager: success
I20260806 04:56:56.193536 5007 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8: acquiring CA information for follower catalog manager: success
I20260806 04:56:56.195432 5007 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 048c9b81ba774b9294bf5b4a0f4cc7b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 04:56:56.212172 4918 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:56:56.210415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33584 (local address 127.13.148.189:38369)
0806 04:56:56.210631 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:56.210637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:56.210720 (+ 83us) server_negotiation.cc:407] Connection header received
0806 04:56:56.210963 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:56.210967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:56.211040 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:56.211162 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:56:56.211958 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33584: BlockingRecv error: recv got EOF from 127.0.0.1:33584 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 04:56:56.505880 5057 catalog_manager.cc:1259] Loaded cluster ID: 8dfa27e0d1fc4afb8b948968561a7c1d
I20260806 04:56:56.505981 5057 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f: loading cluster ID for follower catalog manager: success
I20260806 04:56:56.508327 5057 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f: acquiring CA information for follower catalog manager: success
I20260806 04:56:56.510344 5057 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b4469595beaf4b96a4ac50ecd34cb59f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:56:56.741585 5122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:56.734604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.13.148.189:38369)
0806 04:56:56.735239 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:56.735246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:56.735474 (+ 228us) server_negotiation.cc:407] Connection header received
0806 04:56:56.735625 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:56.735630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:56.735838 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:56.735991 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:56.736386 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:56.737839 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:56.738995 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:56.739597 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:56.739725 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:56.740173 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:56.740931 (+ 758us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:56.741073 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:56.741139 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 04:56:56.741232 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":311,"threads_started":1}
I20260806 04:56:56.752743 5128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:56.747601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.13.148.188:43391)
0806 04:56:56.748175 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:56.748183 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:56.748198 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:56:56.748295 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:56.748298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:56.748423 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:56.748588 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:56.748958 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:56.749587 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:56.750708 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:56.751058 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:56.751176 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:56.751667 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:56.752158 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:56.752310 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:56.752391 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 04:56:56.752470 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":356,"threads_started":1}
I20260806 04:56:56.766371 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:56.757994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48544 (local address 127.13.148.190:36783)
0806 04:56:56.758487 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:56.758493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:56.759019 (+ 526us) server_negotiation.cc:407] Connection header received
0806 04:56:56.759347 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:56.759354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:56.759554 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:56.759698 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:56.760338 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:56.761300 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:56.763409 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:56.764010 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:56.764161 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:56.764937 (+ 776us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:56.765320 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:56.765422 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:56.766006 (+ 584us) server_negotiation.cc:299] Negotiation successful
0806 04:56:56.766086 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":243,"threads_started":1}
I20260806 04:56:56.767935 5122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:56.758833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.13.148.189:38369)
0806 04:56:56.758976 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:56.758982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:56.759393 (+ 411us) server_negotiation.cc:407] Connection header received
0806 04:56:56.759639 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:56.759646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:56.759719 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:56.759775 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:56.760668 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:56.762007 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:56.764952 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:56.765779 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:56.765885 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:56.766911 (+ 1026us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:56.767076 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:56.767144 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:56.767439 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 04:56:56.767532 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 04:56:56.772691 5128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:56:56.759547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.13.148.188:43391)
0806 04:56:56.759693 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 04:56:56.759697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:56:56.765931 (+ 6234us) server_negotiation.cc:407] Connection header received
0806 04:56:56.766798 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:56:56.766802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:56:56.766891 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:56:56.766955 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:56:56.767630 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:56.768614 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:56.770728 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:56:56.771195 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:56:56.771316 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:56:56.771806 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:56:56.772121 (+ 315us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:56:56.772214 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 04:56:56.772401 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 04:56:56.772467 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:56:56.774685 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 4769
I20260806 04:56:56.775609 4832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:56.885987 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 4769
W20260806 04:56:56.907622 4940 connection.cc:535] server connection from 127.0.0.1:49096 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 04:56:56.908042 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 4844
I20260806 04:56:56.909109 4909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:57.003161 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 4844
I20260806 04:56:57.023784 13906 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task10hk37/build/asan/bin/kudu pid 4921
I20260806 04:56:57.024895 4985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 04:56:57.123483 13906 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10hk37/build/asan/bin/kudu with pid 4921
2026-08-06T04:56:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4658 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9231 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (344102 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST